2025-11-01 13:15:12 [scrapy.utils.log] (PID: 131) INFO: Scrapy 2.12.0 started (bot: catalog_extraction) 2025-11-01 13:15:12 [scrapy.utils.log] (PID: 131) INFO: Versions: lxml 5.3.1.0, libxml2 2.12.9, cssselect 1.3.0, parsel 1.10.0, w3lib 2.3.1, Twisted 24.11.0, Python 3.11.13 (main, Jun 10 2025, 23:54:42) [GCC 12.2.0], pyOpenSSL 25.0.0 (OpenSSL 3.4.1 11 Feb 2025), cryptography 44.0.2, Platform Linux-6.9.12-x86_64-with-glibc2.36 2025-11-01 13:15:12 [international_plastics] (PID: 131) INFO: Starting extraction spider international_plastics... 2025-11-01 13:15:12 [scrapy.addons] (PID: 131) INFO: Enabled addons: [] 2025-11-01 13:15:12 [py.warnings] (PID: 131) WARNING: /usr/local/lib/python3.11/site-packages/scrapy/utils/request.py:120: ScrapyDeprecationWarning: 'REQUEST_FINGERPRINTER_IMPLEMENTATION' is a deprecated setting. It will be removed in a future version of Scrapy. return cls(crawler) 2025-11-01 13:15:12 [scrapy.extensions.telnet] (PID: 131) INFO: Telnet Password: e8b1ecb3fe035516 2025-11-01 13:15:12 [py.warnings] (PID: 131) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 13:15:12 [scrapy.middleware] (PID: 131) INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.closespider.CloseSpider', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats', 'spidermon.contrib.scrapy.extensions.Spidermon'] 2025-11-01 13:15:12 [scrapy.crawler] (PID: 131) INFO: Overridden settings: {'BOT_NAME': 'catalog_extraction', 'CONCURRENT_ITEMS': 250, 'CONCURRENT_REQUESTS': 24, 'FEED_EXPORT_ENCODING': 'utf-8', 'LOG_FILE': '/var/lib/scrapyd/logs/catalog_extraction/international_plastics/c82a6a52b72411f0aeb54200a9fe0102.log', 'LOG_FORMAT': '%(asctime)s [%(name)s] (PID: %(process)d) %(levelname)s: ' '%(message)s', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'catalog_extraction.spiders', 'REQUEST_FINGERPRINTER_CLASS': 'scrapy_poet.ScrapyPoetRequestFingerprinter', 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7', 'SPIDER_MODULES': ['catalog_extraction.spiders'], 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor', 'USER_AGENT': None} 2025-11-01 13:15:13 [scrapy_poet.injection] (PID: 131) INFO: Loading providers: [, , , , , , ] 2025-11-01 13:15:13 [scrapy.middleware] (PID: 131) INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scraping_utils.middlewares.downloaders.ProxyManagerDownloaderMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scraping_utils.middlewares.downloaders.HeadersSpooferDownloaderMiddleware', 'scrapy_poet.InjectionMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy_poet.DownloaderStatsMiddleware'] 2025-11-01 13:15:13 [NotFoundHandlerSpiderMiddleware] (PID: 131) INFO: NotFoundHandlerSpiderMiddleware running on PRODUCTION environment. 2025-11-01 13:15:13 [scrapy.middleware] (PID: 131) INFO: Enabled spider middlewares: ['catalog_extraction.middlewares.NotFoundHandlerSpiderMiddleware', 'catalog_extraction.middlewares.FixtureSavingMiddleware', 'scrapy_poet.RetryMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2025-11-01 13:15:13 [scrapy.middleware] (PID: 131) INFO: Enabled item pipelines: ['catalog_extraction.pipelines.DuplicatedSKUsFilterPipeline', 'catalog_extraction.pipelines.DiscontinuedProductsAdjustmentPipeline', 'catalog_extraction.pipelines.PriceRoundingPipeline', 'scraping_utils.pipelines.AttachSupplierPipeline', 'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline'] 2025-11-01 13:15:13 [scrapy.core.engine] (PID: 131) INFO: Spider opened 2025-11-01 13:15:13 [scrapy.extensions.closespider] (PID: 131) INFO: Spider will stop when no items are produced after 1800 seconds. 2025-11-01 13:15:13 [scrapy.extensions.logstats] (PID: 131) INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2025-11-01 13:15:13 [scrapy.extensions.telnet] (PID: 131) INFO: Telnet console listening on 127.0.0.1:6026 2025-11-01 13:15:15 [ProxyManagerDownloaderMiddleware] (PID: 131) INFO: Using brd-customer-hl_13cda1e4-zone-main_datacenter_proxy as the default proxy for ProxyManagerDownloaderMiddleware. 2025-11-01 13:15:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:16 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:17 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:17 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-12-x-60-tall-boxes-p-bx-121260 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-10-x-8-standard-boxes-p-bx-101008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:19 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-2.1-mil-3-in-x-55-yds-pvc-natural-rubber-p-tprpvc210355 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:19 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/mailing-labels/glass-handle-with-care-mailing-labels-p-sml114 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/36-x-18-x-6-flat-boxes-p-bx-361806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/17-x-17-x-17-cube-boxes-p-bx-171717 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-11-x-6-standard-boxes-p-bx-141106 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-10-x-4-flat-boxes-p-bx-101004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/13-x-11-x-6-standard-boxes-p-bx-131106 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:21 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:21 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-6-x-6-long-boxes-p-bx-240606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:21 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:22 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:22 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/amber-uv-protective-bags/4-x-6-3-mil-amber-reclosable-bags-p-cza30406 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:22 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/13-x-13-x-2-flat-boxes-p-bx-131302 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:23 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-51-x-48-x-85-clear-plastic-pallet-covers-p-pc4514885 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/56-x-10-x-32-bike-moving-boxes-p-bx-561032bb already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-68-x-65-x-82-clear-plastic-pallet-covers-p-pc2686582 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-51-x-49-x-97-clear-plastic-pallet-covers-p-pc4514997 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:23 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/40-x-22-x-72-4-mil-clear-plastic-pallet-covers-p-pc4402272 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/44-x-36-x-88-4-mil-clear-plastic-pallet-covers-p-pc4443688 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/tan-2.1-mil-2-in-x-110-yds-pvc-natural-rubber-p-tprpvc2102110-tan-6 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:24 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-1.9-mil-2-in-x-110-yds-natural-rubber-tape-p-tpr1902110 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/miscellaneous-dangerous-goods-dot-labels-p-hml431 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-6-x-6-long-boxes-p-bx-200606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:26 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shockwatch-labels/shockwatch-2-50g-label-p-si-50000k already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:26 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:26 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/4x4-explosive-1.1a-dot-labels-p-hml432 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2mil-54-x-44-x-76-black-plastic-pallet-covers-p-pcb2544476 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18.5-x-12.5-x-7-standard-boxes-p-bx-181207 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-18-x-16-standard-boxes-p-bx-181816 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-9-x-4-flat-boxes-p-bx-120904 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-12-x-3-flat-boxes-p-bx-181203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:30 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/8x8-4mil-ziplock-bags-p-cz40808 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/3m-600-.75in-72yards-scotch-transparent-tape-p-3mst-600-02 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/15-x-15-x-3-flat-boxes-p-bx-151503 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:31 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:31 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:31 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:31 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:32 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:32 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:32 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/36-x-5-x-36-sideloaded-boxes-p-bx-360536 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/36-x-5-x-24-sideloaded-boxes-p-bx-360524 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/13-x-10-x-5-standard-boxes-p-bx-131005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/6-x-4-x-4-standard-boxes-p-bx-060404 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-10-x-3-flat-boxes-p-bx-121003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/4-x-4-x-4-white-corrugated-boxes-p-bx-040404w already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/11.25-x-8.625-x-10-standard-boxes-p-bx-110810 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:32 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:33 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:33 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:33 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:34 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-51-x-49-x-85-clear-plastic-pallet-covers-p-pc3514985 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-58-x-46-x-96-clear-plastic-pallet-covers-p-pc4584696 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/56-x-8-x-36-tv-moving-boxes-p-bx-560836tv already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:34 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/36-x-32-x-60-4-mil-clear-plastic-pallet-covers-p-pc4363260 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/12-x-12-x-46-lamp-moving-boxes-p-bx-121246lb already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:36 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:36 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/fragile-labels/3x5-fluorescent-yellow-fragile-labels-p-ipm2422 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2mil-50x49x110-clear-plastic-pallet-covers-p-pc991102 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/twisted-handle-shopping-bags/8-x-4-x-10-black-twisted-paper-handle-shoppers-p-mb-thp-8-black already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:37 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:37 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/6x6-4mil-ziplock-bags-p-cz40606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/32-x-28-x-84-4-mil-clear-plastic-pallet-covers-p-pc4322884 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:38 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:39 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:39 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:39 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:39 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:40 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:40 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-18-x-4-flat-boxes-p-bx-241804 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:40 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/6x8-4mil-ziplock-bags-p-cz40608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-6-x-6-long-boxes-p-bx-100606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:42 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:42 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:42 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-8-x-4-standard-boxes-p-bx-120804 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/11-x-8-x-8-standard-boxes-p-bx-110808-2 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-8-x-5-standard-boxes-p-bx-080805 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/13-x-13-x-9-standard-boxes-p-bx-131309 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shockwatch-labels/shockwatch-2-25g-label-p-si-48000k already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-46-x-42-x-72-clear-plastic-pallet-covers-p-pc4464272 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:45 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:46 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:46 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:46 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/3m-600-.5in-72yards-scotch-transparent-tape-p-3mst-600-01 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:46 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:47 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:47 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/60-x-60-1-mil-pallet-top-sheets-p-pts16060 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:48 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/8x10-4mil-ziplock-bags-p-cz40810 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/3x4-4mil-ziplock-bags-p-cz40304 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/infectious-substances-dot-labels-p-hml439 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:50 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:50 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:50 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:50 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3mil-51-x-49-x-97-black-plastic-pallet-covers-p-pcb3514997 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/poly-tubing/16in-6-mil-black-tubing-on-roll-p-ptb616 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/6x12-4mil-ziplock-bags-p-cz40612 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:51 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/fragile-labels/glass-handle-with-care-5x3-warning-labels-p-scl566 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/acrylic-2-in-x-55-yds-2.6-mil-carton-sealing-tape-p-tl-t901291t already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:52 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:52 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-14-x-14-white-corrugated-boxes-p-bx-141414w already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12.5-x-12.5-x-6-flat-boxes-p-bx-121206-2 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-10-x-6-standard-boxes-p-bx-121006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/9-x-6-x-4-standard-boxes-p-bx-090604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-12-x-10-standard-boxes-p-bx-181210 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-10-x-4-flat-boxes-p-bx-241004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/amber-uv-protective-bags/9-x-12-3-mil-amber-reclosable-bags-p-cza30912 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:54 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-12-x-6-flat-boxes-p-bx-161206 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:56 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-48-x-42-x-48-clear-plastic-pallet-covers-p-pc4484248 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-48-x-46-x-72-clear-plastic-pallet-covers-p-pc2484672 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:56 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/18-x-18-x-28-dish-moving-boxes-p-bx-181828dp already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:57 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:57 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:57 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:58 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:58 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:58 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:58 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-46-x-36-x-72-clear-plastic-pallet-covers-p-pc2463672 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/twisted-handle-shopping-bags/8-x-4-x-10-chocolate-brown-twisted-paper-handle-shoppers-p-mb-thp-8-choc already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-51-x-49-x-97-clear-plastic-pallet-covers-p-pc3514997 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/radioactive-i-contents-dot-labels-p-hml416 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:01 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/4-x-4-x-24-tall-boxes-p-bx-040424 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/4-x-4-x-36-tall-boxes-p-bx-040436 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/6-x-6-x-36-tall-boxes-p-bx-060636 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/48-x-46-x-72-pallet-shrink-bags-p-pcs484672 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:03 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/heavy-duty/11.5-x-15.4-x-11.5-heavy-duty-boxes-p-bx-111115keg already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:03 [py.warnings] (PID: 131) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 13:16:04 [scrapy.extensions.feedexport] (PID: 131) INFO: Stored bq feed (1000 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:1 2025-11-01 13:16:04 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:04 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-16-x-8-standard-boxes-p-bx-201608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:05 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-8-x-8-cube-boxes-p-bx-080808 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3mil-51x49x97-black-plastic-pallet-covers-p-pcbl009730 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/twisted-handle-shopping-bags/8-x-4-x-10-oatmeal-twisted-paper-handle-shoppers-p-mb-thp-8-oatmeal already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:05 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shrink-wrap/bundling-film/20in-x-4200ft-2-mil-shrink-bundling-film-p-ps2020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-16-x-8-standard-boxes-p-bx-181608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/5x8-4mil-ziplock-bags-p-cz40508 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:06 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:07 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:07 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:07 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-58-x-40-x-80-clear-plastic-pallet-covers-p-pc3584080 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-58-x-46-x-96-clear-plastic-pallet-covers-p-pc2584696 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/70-x-8-x-42-tv-moving-boxes-p-bx-700842tv already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:08 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:08 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:08 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/heavy-duty/48-x-24-x-12-heavy-duty-corrugated-boxes-p-bx-482412hddw already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/42-x-28-x-48-4-mil-clear-plastic-pallet-covers-p-pc4422848 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-1.9-mil-2-in-x-55-yds-natural-rubber-tape-p-tpr190255 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/72-x-72-1.5-mil-pallet-top-sheets-p-pts157272 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:10 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-48-x-46-x-72-clear-plastic-pallet-covers-p-pc3484672 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/18x20-4mil-ziplock-bags-p-cz41820 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:11 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:11 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/48-x-10-x-10-standard-boxes-p-bx-481010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:11 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:11 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:12 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:12 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2mil-48-x-40-x-100-black-plastic-pallet-covers-p-pcb24840100 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/24-x-24-x-18-deluxe-moving-boxes-p-bx-242418d already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-16-x-6-flat-boxes-p-bx-201606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:13 [scrapy.extensions.logstats] (PID: 131) INFO: Crawled 1217 pages (at 1217 pages/min), scraped 1209 items (at 1209 items/min) 2025-11-01 13:16:13 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:13 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:13 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:13 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:14 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-10-x-10-cube-boxes-p-bx-101010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:14 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-7-x-7-standard-boxes-p-bx-140707 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/heavy-duty/36-x-24-x-36-heavy-duty-boxes-p-bx-363624hddw already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:16 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:16 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/28-x-16-x-7-flat-boxes-p-bx-281607 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/17-x-17-x-6-flat-boxes-p-bx-171706 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/6x4-4mil-ziplock-bags-p-cz40604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:16 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:16 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/newsprint-paper/24-x-36-newsprint-sheets-p-np-2430 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:17 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-51-x-49-x-73-clear-plastic-pallet-covers-p-pc3514973 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-54-x-44-x-96-clear-plastic-pallet-covers-p-pc2544496 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:18 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/32-x-18-x-8-standard-boxes-p-bx-321808 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/53-x-42-x-78-4-mil-clear-plastic-pallet-covers-p-pc4534278 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:19 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-2.1-mil-2-in-x-55-yds-pvc-natural-rubber-p-tprpvc210255 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-8-x-8-standard-boxes-p-bx-160808 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/twisted-handle-shopping-bags/8-x-4-x-10-orange-twisted-paper-handle-shoppers-p-mb-thp-8-orange already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/36-x-36-x-108-4-mil-clear-plastic-pallet-covers-p-pc43636108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/hot-melt-3-in-x-55-yds-2.5-mil-carton-sealing-tape-p-tl-t906900 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:22 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/dry-cleaning-bags/21-x-4-x-72-plastic-dry-cleaning-bags-p-gar-72 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-12-x-10-standard-boxes-p-bx-161210 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:26 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-7-x-3-flat-boxes-p-bx-100703 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-12-x-8-standard-boxes-p-bx-181208 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/amber-uv-protective-bags/8x14-3mil-uv-protective-bags-p-mgm30814 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/13-x-13-x-10-standard-boxes-p-bx-131310 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/13.5-x-13.5-x-7.5-standard-boxes-p-bx-131307-2 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-10-x-6-standard-boxes-p-bx-141006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-5-x-5-long-boxes-p-bx-100505 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-11-x-3-flat-boxes-p-bx-141103 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-14-x-2-flat-boxes-p-bx-141402 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-8-x-12-white-corrugated-boxes-p-bx-121208w already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/4-x-4-x-20-tall-boxes-p-bx-040420 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/50-x-48-x-84-pallet-shrink-bags-p-pcs504884 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-20-x-12-standard-boxes-p-bx-202012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/24-x-13-x-31-suitcase-moving-boxes-p-bx-241331sb already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:30 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-48-x-40-x-100-clear-plastic-pallet-covers-p-pc24840100 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:30 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/30-x-30-x-10-standard-boxes-p-bx-303010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/36-x-36-x-72-4-mil-clear-plastic-pallet-covers-p-pc4363672 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/36-x-36-x-48-4-mil-clear-plastic-pallet-covers-p-pc4363648 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-8-x-4-standard-boxes-p-bx-240804 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:31 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:31 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-1.7-mil-2-in-x-55-yds-natural-rubber-tape-p-tpr170255-6 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:31 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-6-x-6-long-boxes-p-bx-160606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:33 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/international-safe-handling-labels/fragile-shipping-lables-international-3x4-p-ipm301 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:34 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:34 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/acrylic-6-pack-2-in-x-110-yds-2-mil-carton-sealing-tape-p-tl-t9024006pk already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/20-x-8-x-50-guitar-moving-boxes-p-bx-200850gub already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/label-dispensers/18in-table-top-label-dispenser-p-ldsl9518 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:36 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3mil-51-x-49-x-85-black-plastic-pallet-covers-p-pcb3514985 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:37 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:37 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:37 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:38 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-9-x-8-standard-boxes-p-bx-120908 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-2.1-mil-3-in-x-55-yds-pvc-natural-rubber-p-tprpvc210355-6 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-12-x-8-standard-boxes-p-bx-141208 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:40 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/4-x-4-x-12-tall-boxes-p-bx-040412 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-18-x-14-standard-boxes-p-bx-181814 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/6-x-6-x-6-white-corrugated-boxes-p-bx-060606w already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/newsprint-paper/48-x-1440-newsprint-roll-p-npr-4890 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-48-x-48-x-72-clear-plastic-pallet-covers-p-pc3484872 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/1-mil-48-x-48-x-96-clear-plastic-pallet-covers-p-pc1484896 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/36-x-21-x-10-wardrobe-moving-boxes-p-bx-362110w already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:43 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14.5-x-8.75-x-12-standard-boxes-p-bx-140812 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:46 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shockwatch-labels/shockwatch-37g-p-si-20686 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/60-x-60-1.5-mil-black-pallet-top-sheets-p-pts156060-b already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/international-safe-handling-labels/3x4-do-not-tumble-labels-p-ipm311 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-14-x-12-standard-boxes-p-bx-181412 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:46 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:47 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:48 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:48 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-10-x-6-flat-boxes-p-bx-201006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/international-safe-handling-labels/keep-dry-shipping-lables-international-3x4-p-ipm303 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:49 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/4-x-4-x-8-standard-boxes-p-bx-040408 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/poly-tubing/10in-6-mil-black-tubing-on-roll-p-ptb610 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:50 [py.warnings] (PID: 131) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 13:16:50 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-20-x-10-standard-boxes-p-bx-202010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shrink-wrap/bundling-film/18in-x-4200ft-2-mil-shrink-bundling-film-p-ps2018 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:51 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/newsprint-paper/18-x-24-newsprint-sheets-p-np-1830 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:51 [scrapy.extensions.feedexport] (PID: 131) INFO: Stored bq feed (1000 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:2 2025-11-01 13:16:51 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:51 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:51 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/7-x-7-x-7-cube-boxes-p-bx-070707 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-16-x-8-standard-boxes-p-bx-161608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:52 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:52 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-10-x-10-standard-boxes-p-bx-141010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:52 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/48-x-12-x-12-long-boxes-p-bx-481212 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-18-x-10-standard-boxes-p-bx-181810 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:52 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/twisted-handle-shopping-bags/8-x-4-x-10-red-twisted-paper-handle-shoppers-p-mb-thp-8-red already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/15-x-15-x-12-standard-boxes-p-bx-151512 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:53 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:53 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:53 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:54 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-26-x-24-x-48-clear-plastic-pallet-covers-p-pc3262448 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-48-x-48-x-84-clear-plastic-pallet-covers-p-pc3484884 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-36-x-24-x-43-clear-plastic-pallet-covers-p-pc2362443 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:55 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:55 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:55 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:55 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:55 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/tan-2.1-mil-2-in-x-55-yds-pvc-natural-rubber-p-tprpvc210255-tan-6 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-20-x-2-flat-boxes-p-bx-202002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shrink-wrap/bundling-film/16-x-2750-poly-shrink-film-p-ps2016t already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:57 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:57 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shockwatch-labels/shockwatch-50g-p-si-20500 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:57 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:57 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-24-x-12-standard-boxes-p-bx-242412 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:58 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-48-x-36-x-80-clear-plastic-pallet-covers-p-pc2483680 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/twisted-handle-shopping-bags/8-x-4-x-10-yellow-twisted-paper-handle-shoppers-p-mb-thp-8-yellow already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/13-x-10-x-4-flat-boxes-p-bx-131004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:59 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/22-x-6-x-16-tv-moving-boxes-p-bx-220616tv already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/4-x-4-x-60-tall-boxes-p-bx-040460 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:59 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:59 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:59 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:01 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-12-x-10-standard-boxes-p-bx-241210 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:01 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:01 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-10-x-30-tall-boxes-p-bx-101030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/13-x-10-x-10-standard-boxes-p-bx-131010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12.5-x-12.5-x-12-standard-boxes-p-bx-121212-2 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-12-x-6-standard-boxes-p-bx-141206 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:03 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/40-x-18-x-8-standard-boxes-p-bx-401808 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/4-x-4-x-28-tall-boxes-p-bx-040428 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:03 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/amber-uv-protective-bags/4x6-3mil-uv-protective-bags-p-mgm30406 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:05 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-48-x-48-x-96-clear-plastic-pallet-covers-p-pc4484896 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:06 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-30-x-26-x-48-clear-plastic-pallet-covers-p-pc3302648 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/1-mil-54-x-52-x-60-clear-plastic-pallet-covers-p-pc1545260 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:07 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:07 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/52-x-43-x-70-4-mil-clear-plastic-pallet-covers-p-pc4524370 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/32-x-28-x-96-4-mil-clear-plastic-pallet-covers-p-pc4322896 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:07 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-48-x-46-x-72-clear-plastic-pallet-covers-p-pc4484672 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:08 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:08 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:08 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shockwatch-labels/shockwatch-2-5g-label-p-si-45000k already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/14x20-4mil-ziplock-bags-p-cz41420 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/13x18-4mil-ziplock-bags-p-cz41318 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:09 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:10 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:10 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/18-x-6-x-45-guitar-moving-boxes-p-bx-180645gub already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/24-x-18-x-24-deluxe-moving-boxes-p-bx-241824d already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/15-x-15-x-4-flat-boxes-p-bx-151504 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:11 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:11 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:11 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:12 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:12 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-12-x-12-cube-boxes-p-bx-121212 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:12 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/label-dispensers/6in-table-top-label-dispenser-p-ldsl9506 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3mil-48-x-42-x-66-black-plastic-pallet-covers-p-pcb3484266 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/2.5x3-4mil-ziplock-bags-p-cz40253 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:13 [scrapy.extensions.logstats] (PID: 131) INFO: Crawled 2488 pages (at 1271 pages/min), scraped 2485 items (at 1276 items/min) 2025-11-01 13:17:13 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:14 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/9-x-8-x-6-standard-boxes-p-bx-090806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-12-x-12-standard-boxes-p-bx-161212 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/36-x-36-x-12-standard-boxes-p-bx-363612 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-10-x-60-tall-boxes-p-bx-101060 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-6-x-4-standard-boxes-p-bx-240604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/13x15-4mil-ziplock-bags-p-cz41315 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/newsprint-paper/12-x-1440-newsprint-roll-p-npr-1290 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:18 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:18 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/7-x-6-x-4-standard-boxes-p-bx-070604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:18 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/54-x-44-x-120-4-mil-clear-plastic-pallet-covers-p-pc45444120 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/58-x-46-x-125-4-mil-clear-plastic-pallet-covers-p-pc45846125 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/50-x-44-x-57-4-mil-clear-plastic-pallet-covers-p-pc4504457 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/42-x-32-x-72-4-mil-clear-plastic-pallet-covers-p-pc4423272 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:19 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:20 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:20 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:20 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:20 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/poison-gas-dot-labels-p-hml414 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:20 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:20 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shockwatch-labels/shockwatch-75g-p-si-20300 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:21 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:21 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/4x4-explosive-1.3c-dot-labels-p-hml434 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/mailing-labels/do-not-bend-mailing-labels-p-sml107 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/14-x-4-x-42-rifle-moving-boxes-p-bx-140442rb already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:22 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:23 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:23 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:24 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tiltwatch/tiltwatch-xtr-labels-200-p-tw-cl-26102 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:24 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3mil-48-x-46-x-72-black-plastic-pallet-covers-p-pcb3484672 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/26-x-12-x-12-standard-boxes-p-bx-261212 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-16-x-14-standard-boxes-p-bx-181614 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-10-x-8-standard-boxes-p-bx-121008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-8-x-5-standard-boxes-p-bx-100805 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/11-x-11-x-9-standard-boxes-p-bx-111109 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/heavy-duty/9-x-13-x-9-heavy-duty-boxes-p-bx-090913keg already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-9-x-6-standard-boxes-p-bx-120906 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/36-x-10-x-10-long-boxes-p-bx-361010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/30-x-24-x-12-standard-boxes-p-bx-302412 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-8-x-3-flat-boxes-p-bx-100803 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/6-x-6-x-5-standard-boxes-p-bx-060605 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/label-dispensers/wall-mount-12.5in-label-dispenser-p-ldm1250 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shockwatch-labels/shockwatch-2-37g-label-p-si-49000k already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:30 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-48-x-46-x-96-clear-plastic-pallet-covers-p-pc2484696 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-44-x-36-x-96-clear-plastic-pallet-covers-p-pc2443696 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/54-x-8-x-28-bike-moving-boxes-p-bx-540828bb already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:30 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/5-x-5-x-8-standard-boxes-p-bx-050508 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:31 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/30-x-24-x-10-standard-boxes-p-bx-302410 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-5-x-4-standard-boxes-p-bx-120504 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/36-x-36-x-60-4-mil-clear-plastic-pallet-covers-p-pc4363660 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/36-x-28-x-60-4-mil-clear-plastic-pallet-covers-p-pc4362860 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/tan-1.9-mil-2-in-x-55-yds-natural-rubber-tape-p-tpr190255-tan-6 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/40-x-10-x-10-standard-boxes-p-bx-401010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:32 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:33 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:33 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3mil-51x49x85-black-plastic-pallet-covers-p-pcbl008530 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shockwatch-labels/shockwatch-2-10g-label-p-si-46000k already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/4x6-4mil-ziplock-bags-p-cz40406 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:34 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/newsprint-paper/36-x-1440-newsprint-roll-p-npr-3690 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/4x4-explosive-1.4g-dot-labels-p-hml440 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/36-x-32-x-48-4-mil-clear-plastic-pallet-covers-p-pc4363248 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:34 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:36 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/10.5-x-6.5-x-57-moving-ski-boxes-p-bx-106257skib already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:36 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:36 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:36 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:37 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/international-safe-handling-labels/3x4-fragile-glass-arrows-labels-p-ipm319 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:37 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:37 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-10-x-40-tall-boxes-p-bx-101040 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/heavy-duty/18-x-8-x-12-heavy-duty-boxes-p-bx-181208hddw already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/international-safe-handling-labels/3x4-do-not-use-blades-to-open-labels-p-ipm325 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:38 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:38 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/heavy-duty/14-x-14-x-14-heavy-duty-boxes-p-bx-141414hddw already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:38 [py.warnings] (PID: 131) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/label-dispensers/wall-mount-3in-label-dispenser-p-ldm300 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-12-x-4-flat-boxes-p-bx-201204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:39 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-16-x-10-standard-boxes-p-bx-161610 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:39 [scrapy.extensions.feedexport] (PID: 131) INFO: Stored bq feed (1000 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:3 2025-11-01 13:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/10x12-4mil-ziplock-bags-p-cz41012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:39 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/22-x-14-x-12-standard-boxes-p-bx-221412 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:40 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:40 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shrink-wrap/bundling-film/16in-x-4200ft-2-mil-shrink-bundling-film-p-ps2016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/15-x-12-x-5-flat-boxes-p-bx-151205 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:40 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-42-x-32-x-72-clear-plastic-pallet-covers-p-pc2423272 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/36-x-36-3-mil-pallet-top-sheets-p-pts33636 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:42 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:42 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/38-x-8-x-26-tv-moving-boxes-p-bx-380826tv already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/24-x-20-x-34-wardrobe-moving-boxes-p-bx-242034w already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/24-x-22-x-60-wardrobe-moving-boxes-p-bx-242260w already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-5-x-5-standard-boxes-p-bx-120505 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/54-x-44-x-96-4-mil-clear-plastic-pallet-covers-p-pc4544496 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-2.1-mil-2-in-x-55-yds-pvc-natural-rubber-p-tprpvc210255-6 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/newsprint-paper/24-x-1440-newsprint-roll-p-npr-2490 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:43 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:43 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:44 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:44 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:44 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-12-x-3-flat-boxes-p-bx-14123r already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-13-x-13-standard-boxes-p-bx-161313 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:44 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-4-x-4-standard-boxes-p-bx-200404 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/72-x-72-1-mil-pallet-top-sheets-p-pts17272 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/12x12-4mil-ziplock-bags-p-cz41212 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/60-x-60-1.25-mil-pallet-top-sheets-p-pts1256060 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/4x4-explosive-1.4s-dot-labels-p-hml444 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/corrosive-dot-labels-p-hml419 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/24-x-18-x-18-deluxe-moving-boxes-p-bx-241818d already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/26-x-20-x-16-standard-boxes-p-bx-262016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:47 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:47 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-14-x-6-flat-boxes-p-bx-241406 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-12-x-4-flat-boxes-p-bx-161204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-16-x-4-flat-boxes-p-bx-241604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/2x3-4mil-ziplock-bags-p-cz40203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/amber-uv-protective-bags/2x9-3mil-uv-protective-bags-p-mgm30209 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:48 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/17.25-x-11.25-x-6-standard-boxes-p-bx-171106 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-12-x-4-flat-boxes-p-bx-141204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/international-safe-handling-labels/perishable-lables-international-3x4-p-ipm313 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3mil-48-x-42-x-48-black-plastic-pallet-covers-p-pcb3484248 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:49 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:49 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:49 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/fragile-labels/3x5-fragile-this-side-up-labels-p-scl521 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/5-x-5-x-5-cube-boxes-p-bx-050505 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-16-x-16-standard-boxes-p-bx-201616 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:51 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-16-x-8-standard-boxes-p-bx-241608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-8-x-8-long-boxes-p-bx-180808 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-16-x-14-standard-boxes-p-bx-201614 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/30-x-6-x-6-long-boxes-p-bx-300606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/24-x-20-x-46-wardrobe-box-bar-p-bx-242046mb-bar already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:52 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/26-x-26-x-36-standard-boxes-p-bx-262636 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-12-x-9-standard-boxes-p-bx-181209-2 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:52 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-12-x-48-tall-boxes-p-bx-121248 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:53 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-48-x-42-x-66-clear-plastic-pallet-covers-p-pc4484266 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-50-x-46-x-86-clear-plastic-pallet-covers-p-pc3504686 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-55-x-45-x-75-clear-plastic-pallet-covers-p-pc4554575 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:55 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/36-x-21-x-20-wardrobe-moving-boxes-p-bx-362120w already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2mil-51x49x73-black-plastic-pallet-covers-p-pcbl007320 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:56 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:56 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/fragile-labels/fragile-4x4-warning-labels-p-scl501 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:57 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/1-mil-48-x-42-x-66-clear-plastic-pallet-covers-p-pc1484266 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:58 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3mil-50x46x86-clear-plastic-pallet-covers-p-pc968630 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-48-x-40-x-72-clear-plastic-pallet-covers-p-pc3484072 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:58 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/flammable-solid-dot-labels-p-hml407 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:59 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:59 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/4-x-4-x-48-tall-boxes-p-bx-040448 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/dry-cleaning-bags/21-x-4-x-30-plastic-dry-cleaning-bags-p-gar-30 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:01 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/poly-tubing/14in-6-mil-black-tubing-on-roll-p-ptb614 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:01 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-12-x-6-standard-boxes-p-bx-181206 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/poly-tubing/24in-6-mil-black-tubing-on-roll-p-ptb624 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:01 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:01 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/6-x-6-x-10-standard-boxes-p-bx-060610 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:01 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:02 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:02 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:02 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:02 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:02 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-42-x-42-x-72-clear-plastic-pallet-covers-p-pc4424272 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:03 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:03 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:03 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/heavy-duty/16-x-16-x-16-heavy-duty-boxes-p-bx-161616hddw already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-18-x-4-flat-boxes-p-bx-181804 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-16-x-4-flat-boxes-p-bx-161604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/9-x-9-x-7-standard-boxes-p-bx-090907 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:06 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-52-x-48-x-130-clear-plastic-pallet-covers-p-pc35248130 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-52-x-44-x-96-clear-plastic-pallet-covers-p-pc2524496 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/18-x-18-x-24-deluxe-moving-boxes-p-bx-181824d already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:07 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/9-x-5-x-4-standard-boxes-p-bx-090504 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:07 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:07 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/50-x-48-x-84-4-mil-clear-plastic-pallet-covers-p-pc4504884 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:07 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:08 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:09 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:09 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/10x13-4mil-ziplock-bags-p-cz41013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/twisted-handle-shopping-bags/8-x-4-x-10-lime-green-twisted-paper-handle-shoppers-p-mb-thp-8-lime already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/36-x-14-x-14-standard-boxes-p-bx-361414 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-32-x-28-x-72-clear-plastic-pallet-covers-p-pc2322872 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/4x4-explosive-1.4b-dot-labels-p-hml435 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-12-x-8-standard-boxes-p-bx-161208 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/20-x-20-x-45-wardrobe-moving-boxes-p-bx-202045w already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:11 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:11 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-14-x-6-flat-boxes-p-bx-201406 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:12 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-14-x-4-flat-boxes-p-bx-241404 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tiltwatch/tiltwatch-xtr-plus-p-tw-24114 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/poly-tubing/4in-6-mil-black-tubing-on-roll-p-ptb604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:13 [scrapy.extensions.logstats] (PID: 131) INFO: Crawled 3706 pages (at 1218 pages/min), scraped 3699 items (at 1214 items/min) 2025-11-01 13:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/36-x-12-x-6-flat-boxes-p-bx-361206 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:13 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:14 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/fragile-labels/4x4-glass-labels-p-scl506 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:16 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:16 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/1-mil-48-x-46-x-72-clear-plastic-pallet-covers-p-pc1484672 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-54-x-44-x-96-clear-plastic-pallet-covers-p-pc3544496 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-58-x-48-x-90-clear-plastic-pallet-covers-p-pc2584890 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/58-x-10-x-10-golf-moving-boxes-p-bx-481010gb already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:18 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/42-x-32-x-84-4-mil-clear-plastic-pallet-covers-p-pc4423284 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/46-x-44-x-80-4-mil-clear-plastic-pallet-covers-p-pc4464480 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:19 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:19 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:19 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:19 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:19 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/72-x-72-2-mil-pallet-top-sheets-p-pts27272 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/hot-melt-6-pack-2-in-x-1000-yds-1.6-mil-carton-sealing-tape-p-tl-t903600 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:21 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:22 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:22 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:22 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/28-x-6-x-20-tv-moving-boxes-p-bx-280620tv already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/14-x-4-x-52-rifle-moving-boxes-p-bx-140452rb already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:22 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:22 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:23 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:23 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tiltwatch/tiltwatch-xtr-labels-500-p-tw-cl-26103 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/14x24-4mil-ziplock-bags-p-cz41424 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/poly-tubing/8in-6-mil-black-tubing-on-roll-p-ptb608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:24 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:24 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:24 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/6-x-6-x-30-tall-boxes-p-bx-060630 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-8-x-3-flat-boxes-p-bx-080803 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-2.2-mil-3-in-x-110-yds-natural-rubber-tape-p-tpr2203110 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/tan-1.9-mil-2-in-x-110-yds-natural-rubber-tape-p-tpr1902110-tan-6 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/52-x-43-x-70-pallet-shrink-bags-p-pcs524370 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/2in-x-110yds-clear-acrylic-tape-carton-sealing-p-cs20211cb already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:26 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-18-x-18-cube-boxes-p-bx-181818 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:28 [py.warnings] (PID: 131) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 13:18:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/6-x-6-x-6-cube-boxes-p-bx-060606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/9-x-7-x-7-standard-boxes-p-bx-090707 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/17-x-14-x-5-flat-boxes-p-bx-171405 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:28 [scrapy.extensions.feedexport] (PID: 131) INFO: Stored bq feed (1000 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:4 2025-11-01 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shockwatch-labels/shockwatch-2-75g-label-p-si-51000k already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/combo-wardrobe-box-and-bar-p-bx-wardcombo already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-68-x-65-x-82-clear-plastic-pallet-covers-p-pc4686582 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-48-x-46-x-96-clear-plastic-pallet-covers-p-pc4484696 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-8-x-12-standard-boxes-p-bx-100812 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:31 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/12-x-12-x-40-lamp-moving-boxes-p-bx-121240lb already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/48-x-30-x-60-4-mil-clear-plastic-pallet-covers-p-pc4483060 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/32-x-28-x-72-4-mil-clear-plastic-pallet-covers-p-pc4322872 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:33 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:33 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/hot-melt-3-in-x-110-yds-2.5-mil-carton-sealing-tape-p-tl-t905900 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/harmful-dot-labels-p-hml428 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-10-x-6-white-corrugated-boxes-p-bx-121006w already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-8-x-30-tall-boxes-p-bx-080830 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/spontaneeously-combustible-dot-labels-p-hml408 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:37 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-12-x-4-flat-boxes-p-bx-181204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:38 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:38 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:38 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:39 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-8-x-4-flat-boxes-p-bx-080804 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:39 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:39 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/9-x-8-x-8-standard-boxes-p-bx-090808 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/amber-uv-protective-bags/3x5-3mil-uv-protective-bags-p-mgm30305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:40 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:40 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/11.25-x-8.75-x-12-standard-boxes-p-bx-110812 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-14-x-12-standard-boxes-p-bx-161412 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/amber-uv-protective-bags/3-x-5-3-mil-amber-reclosable-bags-p-cza30305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/7-x-7-x-5-standard-boxes-p-bx-070705 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/28-x-16-x-10-standard-boxes-p-bx-281610 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/amber-uv-protective-bags/2x3-3mil-uv-protective-bags-p-mgm30203 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:42 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-48-x-42-x-66-clear-plastic-pallet-covers-p-pc2484266 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:43 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-42-x-42-x-96-clear-plastic-pallet-covers-p-pc2424296 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/1-mil-36-x-27-x-65-clear-plastic-pallet-covers-p-pc1362765 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2mil-48-x-42-x-48-black-plastic-pallet-covers-p-pcb2484248 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-36-x-28-x-52-clear-plastic-pallet-covers-p-pc3362852 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:43 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:43 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:43 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:44 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/heavy-duty/24-x-18-x-24-heavy-duty-corrugated-boxes-p-bx-241824hddw already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/32-x-28-x-60-4-mil-clear-plastic-pallet-covers-p-pc4322860 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-54-x-44-x-120-clear-plastic-pallet-covers-p-pc25444120 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/6x10-4mil-ziplock-bags-p-cz40610 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:45 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:45 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:45 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:46 [scrapy.core.scraper] (PID: 131) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 83, in _process_spider_input result = method(response=response, spider=spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/middlewares.py", line 234, in process_spider_input raise ProductNotFound(f"Page {response.url} returned 404 status code.") scraping_utils.common.exceptions.ProductNotFound: Page https://www.interplas.com/trash-bags/green-biodegradable-garbage-bags-p-cl-big-4761 returned 404 status code. 2025-11-01 13:18:46 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:46 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:46 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/32-x-32-x-12-standard-boxes-p-bx-323212 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/twisted-handle-shopping-bags/8-x-4-x-10-forest-green-twisted-paper-handle-shoppers-p-mb-thp-8-forest already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/flammable-gas-dot-labels-p-hml404 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:47 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:47 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:47 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/15-x-12-x-10-standard-boxes-p-bx-151210 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:48 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:48 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:48 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:48 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-14-x-14-cube-boxes-p-bx-141414 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-10-x-38-tall-boxes-p-bx-101038 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/amber-uv-protective-bags/12x12-3mil-uv-protective-bags-p-mgm31212 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:50 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-6-x-3-flat-boxes-p-bx-100603 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:51 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/newsprint-paper/24-x-30-newsprint-sheets-p-np-2433 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shrink-wrap/bundling-film/28-x-2500-polyethylene-shrink-film-p-ps2028 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:51 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/newsprint-paper/20-x-30-newsprint-sheets-p-np-2030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/newsprint-paper/36-x-48-newsprint-sheets-p-np-3630 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:52 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:52 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:52 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:52 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/17-x-14-x-14-standard-boxes-p-bx-171414 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-10-x-8-standard-boxes-p-bx-141008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/amber-uv-protective-bags/6-x-8-3-mil-amber-reclosable-bags-p-cza30608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/heavy-duty/36-x-36-x-36-heavy-duty-boxes-p-bx-363636hddw already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/polypropylene-zip-lock-bags/3-x-5-2-mil-polypropylene-hang-hole-zipper-lock-bags-p-czpph20305 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:53 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/9-x-6-x-5-standard-boxes-p-bx-090605 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/international-safe-handling-labels/do-not-stack-shipping-lables-international-3x4-p-ipm309 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-10-x-10-standard-boxes-p-bx-201010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-10-x-8-standard-boxes-p-bx-161008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/15-x-10-x-4-flat-boxes-p-bx-151004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-2.2-mil-2-in-x-55-yds-natural-rubber-tape-p-tpr220255 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:54 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/44-x-44-x-70-pallet-shrink-bags-p-pcs444470 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:55 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-54-x-44-x-72-clear-plastic-pallet-covers-p-pc4544472 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:55 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-41-x-31-x-72-clear-plastic-pallet-covers-p-pc2413172 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-70-x-44-x-62-clear-plastic-pallet-covers-p-pc3704462 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-51-x-49-x-73-clear-plastic-pallet-covers-p-pc4514973 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-58-x-43-x-76-clear-plastic-pallet-covers-p-pc2484376 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:56 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-14-x-16-standard-boxes-p-bx-141416 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-24-x-9-standard-boxes-p-bx-242409 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/46-x-44-x-100-4-mil-clear-plastic-pallet-covers-p-pc44644100 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/52-x-48-x-120-4-mil-clear-plastic-pallet-covers-p-pc45248120 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:57 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/poison-dot-labels-p-hml415 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:58 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2mil-50x42x69-black-plastic-pallet-covers-p-pcb926920 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:59 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:59 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:59 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:59 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/4x8-4mil-ziplock-bags-p-cz40408 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:59 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-6-x-8-standard-boxes-p-bx-120608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3mil-52-x-44-x-60-black-plastic-pallet-covers-p-pcb3524460 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:02 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3mil-51-x-49-x-73-black-plastic-pallet-covers-p-pcb3514973 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/acrylic-2-in-x-110-yds-2-mil-carton-sealing-tape-p-tl-t902400 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-2.2-mil-3-in-x-110-yds-natural-rubber-tape-p-tpr2203110-6 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-10-x-24-tall-boxes-p-bx-101024 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:03 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/6-x-6-x-48-tall-boxes-p-bx-060648 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/newsprint-paper/24-x-36-newsprint-sheets-p-np-243010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:04 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/amber-uv-protective-bags/8x8-3mil-uv-protective-bags-p-mgm30808 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-10-x-10-standard-boxes-p-bx-121010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-12-x-6-flat-boxes-p-bx-121206 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/36-x-16-x-16-standard-boxes-p-bx-361616 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-6-x-6-long-boxes-p-bx-180606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/7-x-7-x-6-standard-boxes-p-bx-070706 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-8-x-6-standard-boxes-p-bx-080806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:05 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/mailing-labels/fragile-handle-with-care-mailing-labels-p-sml115 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:06 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-8-x-42-tall-boxes-p-bx-080842 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-14-x-8-standard-boxes-p-bx-181408 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/17-x-17-x-4-flat-boxes-p-bx-171704 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:06 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:06 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-8-x-20-tall-boxes-p-bx-080820 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:07 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/dry-cleaning-bags/21-x-4-x-54-plastic-dry-cleaning-bags-p-gar-54 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:07 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:07 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:08 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:08 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:08 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/40-x-12-x-12-golf-moving-boxes-p-bx-401212gb already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/12-x-12-x--deluxe-moving-boxes-p-bx-121212d already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-48-x-48-x-72-clear-plastic-pallet-covers-p-pc4484872 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/1.25mil-54x44x96-clear-plastic-pallet-covers-p-pc1544496 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:09 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:10 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/1-mil-48-x-42-x-48-clear-plastic-pallet-covers-p-pc1484248 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:10 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:10 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/22-x-14-x-14-standard-boxes-p-bx-221414 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:10 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-14-x-4-flat-boxes-p-bx-181404 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/12x15-4mil-ziplock-bags-p-cz41215 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:13 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:13 [scrapy.extensions.logstats] (PID: 131) INFO: Crawled 4955 pages (at 1249 pages/min), scraped 4944 items (at 1245 items/min) 2025-11-01 13:19:13 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:13 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-6-x-6-long-boxes-p-bx-140606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:13 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/20-x-8-x-60-keyboard-moving-boxes-p-bx-200860kb already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:14 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:15 [py.warnings] (PID: 131) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 13:19:16 [scrapy.extensions.feedexport] (PID: 131) INFO: Stored bq feed (1000 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:5 2025-11-01 13:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shrink-wrap/bundling-film/24in-x-4200ft-2-mil-shrink-bundling-film-p-ps2024 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:17 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:17 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:17 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/11-x-11-x-3-flat-boxes-p-bx-111103 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/twisted-handle-shopping-bags/8-x-4-x-10-pink-twisted-paper-handle-shoppers-p-mb-thp-8-pink already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:18 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:18 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/9-x-9-x-9-cube-boxes-p-bx-090909 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-16-x-16-cube-boxes-p-bx-161616 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:19 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-9-x-2-flat-boxes-p-bx-120902 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-14-x-6-flat-boxes-p-bx-161406 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/amber-uv-protective-bags/6x8-3mil-uv-protective-bags-p-mgm30608 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/7-x-7-x-12-standard-boxes-p-bx-070712 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:19 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:19 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-54-x-52-x-60-clear-plastic-pallet-covers-p-pc4545260 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-54-x-44-x-72-clear-plastic-pallet-covers-p-pc2544472 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:21 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:22 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:22 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/5-x-5-x-3-standard-boxes-p-bx-050503 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/7-x-5-x-3-standard-boxes-p-bx-070503 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/9-x-4-x-4-standard-boxes-p-bx-090404 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:23 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:23 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:23 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/44-x-36-x-108-4-mil-clear-plastic-pallet-covers-p-pc44436108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:23 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:24 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/26-x-20-x-4-flat-boxes-p-bx-262004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:24 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/28-x-12-x-6-flat-boxes-p-bx-281206 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:24 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/60-x-60-1.5-mil-pallet-top-sheets-p-pts156060 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:24 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/twisted-handle-shopping-bags/8-x-4-x-10-sage-green-twisted-paper-handle-shoppers-p-mb-thp-8-sage already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-52-x-48-x-96-clear-plastic-pallet-covers-p-pc2524896 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/twisted-handle-shopping-bags/8-x-4-x-10-purple-twisted-paper-handle-shoppers-p-mb-thp-8-purple already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-1.7-mil-3-in-x-110-yds-natural-rubber-tape-p-tpr1703110-6 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-1.7-mil-3-in-x-110-yds-natural-rubber-tape-p-tpr1703110 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/acrylic-3-in-x-55-yds-2.6-mil-carton-sealing-tape-p-tl-t905291 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:26 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:26 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-6-x-4-long-boxes-p-bx-140604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/17.25-x-11.25-x-8-standard-boxes-p-bx-171108 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/newsprint-paper/24-x-36-newsprint-sheets-p-np-243025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/label-dispensers/wall-mount-4.5in-label-dispenser-p-ldm450 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:30 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:30 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-20-x-16-standard-boxes-p-bx-242016 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/label-dispensers/wall-mount-8.5in-label-dispenser-p-ldm850 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-4-x-4-long-boxes-p-bx-120404 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-24-x-24-cube-boxes-p-bx-242424 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-5-x-5-standard-boxes-p-bx-080505 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/amber-uv-protective-bags/2.5-x-9-3-mil-amber-reclosable-bags-p-cza32509 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-15-x-6-flat-boxes-p-bx-201506 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-12-x-4-flat-boxes-p-bx-241204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/13-x-9-x-4-flat-boxes-p-bx-13904r already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:31 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:31 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/international-safe-handling-labels/handle-with-care-shipping-lables-international-3x4-p-ipm302 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/7x9-4mil-ziplock-bags-p-cz40709 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:32 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:32 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:33 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:33 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-52-x-44-x-96-clear-plastic-pallet-covers-p-pc3524496 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:33 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-52-x-48-x-96-clear-plastic-pallet-covers-p-pc4524896 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:33 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:34 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-5-x-4-standard-boxes-p-bx-100504 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:34 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-1.9-mil-2-in-x-110-yds-natural-rubber-tape-p-tpr1902110-6 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:36 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:36 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:36 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/fragile-labels/do-not-crush-5x3-warning-labels-p-scl584 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/newsprint-paper/30-x-1440-newsprint-roll-p-npr-3090 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:37 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/dangerous-when-wet-dot-labels-p-hml409 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:37 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:37 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/fragile-labels/3x5-fragile-handle-with-care-labels-p-ipm1070 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:38 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/14-x-8-x-6-standard-boxes-p-bx-140806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/dry-cleaning-bags/21-x-4-x-60-plastic-dry-cleaning-bags-p-gar-60 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/label-dispensers/12in-table-top-label-dispenser-p-ldsl9512 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/15-x-12-x-6-standard-boxes-p-bx-151206 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/poly-tubing/6in-6-mil-black-tubing-on-roll-p-ptb606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:40 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:40 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/50-x-44-x-57-pallet-shrink-bags-p-pcs504457 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3mil-48-x-40-x-48-black-plastic-pallet-covers-p-pcb3484048 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-8-x-8-standard-boxes-p-bx-120808 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/13-x-11-x-11-standard-boxes-p-bx-131111 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-6-x-4-long-boxes-p-bx-120604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-6-x-4-standard-boxes-p-bx-080604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/26-x-20-x-20-standard-boxes-p-bx-262020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-6-x-3-flat-boxes-p-bx-080603 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:43 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-7-x-7-standard-boxes-p-bx-100707 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:44 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-2.2-mil-2-in-x-55-yds-natural-rubber-tape-p-tpr220255-6 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:45 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:45 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-52-x-48-x-60-clear-plastic-pallet-covers-p-pc3524860 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/36-x-8-x-4-standard-boxes-p-bx-360804 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/48-x-36-x-72-4-mil-clear-plastic-pallet-covers-p-pc4483672 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-48-x-48-x-96-clear-plastic-pallet-covers-p-pc2484896 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-40-x-24-x-72-clear-plastic-pallet-covers-p-pc2402472 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/tan-1.9-mil-2-in-x-55-yds-natural-rubber-tape-p-tpr190255-tan already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/tan-1.9-mil-2-in-x-110-yds-natural-rubber-tape-p-tpr1902110-tan already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/36-x-36-x-18-standard-boxes-p-bx-363618 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:48 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:48 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:48 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:49 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:49 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:49 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/1-mil-51-x-48-x-75-clear-plastic-pallet-covers-p-pc1514875 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:49 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:49 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:49 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:49 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-60-x-40-x-85-clear-plastic-pallet-covers-p-pc3604085 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/36-x-18-x-12-standard-boxes-p-bx-361812 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:50 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/20-wardrobe-moving-boxes-p-bx-bar20 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:51 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:51 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:52 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:52 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-14-x-4-flat-boxes-p-bx-201404 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:53 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/polypropylene-zip-lock-bags/4-x-6-2-mil-polypropylene-hang-hole-zipper-lock-bags-p-czpph20406 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:53 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:53 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/heavy-duty/8-x-8-x-8-heavy-duty-boxes-p-bx-080808hddw already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:54 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:54 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/60-x-72-1.5-mil-pallet-top-sheets-p-pts156072 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:54 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-7-x-5-standard-boxes-p-bx-100705 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:54 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:55 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:55 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/7-x-7-x-4.5-standard-boxes-p-bx-0707045 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/28-x-12-x-12-standard-boxes-p-bx-281212 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:55 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/polypropylene-zip-lock-bags/10-x-12-2-mil-polypropylene-hang-hole-zipper-lock-bags-p-czpph21012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:56 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:56 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:56 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-10-x-6-standard-boxes-p-bx-161006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/11-x-8-x-6-standard-boxes-p-bx-110806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:57 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/4-mil-44-x-36-x-80-clear-plastic-pallet-covers-p-pc4443680 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:58 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/8-x-4-x-46-rifle-moving-boxes-p-bx-080446rb already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:58 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:58 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:58 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:58 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-6-x-5-standard-boxes-p-bx-120605 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:58 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/30-x-30-x-12-standard-boxes-p-bx-303012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shockwatch-labels/shockwatch-2-companion-labels-p-si-cl-26106 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:59 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:59 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:00 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-13-x-10-standard-boxes-p-bx-161310 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:00 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:00 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/60-x-60-3-mil-pallet-top-sheets-p-pts36060 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:01 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:01 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:02 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:02 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:03 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:03 [py.warnings] (PID: 131) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 13:20:04 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:04 [scrapy.extensions.feedexport] (PID: 131) INFO: Stored bq feed (1000 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:6 2025-11-01 13:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/4-x-4-x-30-tall-boxes-p-bx-040430 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-12-x-12-white-corrugated-boxes-p-bx-181212w already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:05 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:05 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:05 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:05 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:06 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:06 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/newsprint-paper/15-x-20-newsprint-sheets-p-np-1530 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:06 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/13-x-10-x-6-standard-boxes-p-bx-131006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:06 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:07 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-6-x-12-white-corrugated-boxes-p-bx-121206w already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:07 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:07 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:07 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:07 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-8-x-6-standard-boxes-p-bx-100806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:08 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:08 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/24-x-20-x-46-wardrobe-boxes-p-bx-242046mb already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:09 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-54-x-44-x-60-clear-plastic-pallet-covers-p-pc2544460 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-51-x-49-x-73-clear-plastic-pallet-covers-p-pc2514973 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2mil-46-x-42-x-68-black-plastic-pallet-covers-p-pcb2464268 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/64-x-8-x-40-tv-moving-boxes-p-bx-640840tv already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:10 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:10 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/28-x-10-x-10-standard-boxes-p-bx-281010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/42-x-32-x-60-4-mil-clear-plastic-pallet-covers-p-pc4423260 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/51-x-49-x-85-4-mil-clear-plastic-pallet-covers-p-pc4514985 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:11 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:11 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/clear-1.9-mil-2-in-x-55-yds-natural-rubber-tape-p-tpr190255-6 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:11 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:11 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-42-x-42-x-72-clear-plastic-pallet-covers-p-pc2424272 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:11 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-18-x-4-flat-boxes-p-bx-201804 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:11 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/26-x-20-x-6-flat-boxes-p-bx-262006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:12 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:13 [scrapy.extensions.logstats] (PID: 131) INFO: Crawled 6215 pages (at 1260 pages/min), scraped 6202 items (at 1258 items/min) 2025-11-01 13:20:13 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:13 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/48-x-40-x-36-standard-boxes-p-bx-484036 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:13 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:13 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:13 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:14 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:14 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:14 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:14 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:15 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:15 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/19-x-12-x-4-flat-boxes-p-bx-191204 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:15 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/10-x-8-x-4-flat-boxes-p-bx-100804 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:16 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:16 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/heavy-duty/9-x-6-x-9-heavy-duty-boxes-p-bx-090906keg already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:17 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shockwatch-labels/shockwatch-2-companion-labels-p-si-cl-26126 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:17 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:17 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-8-x-16-standard-boxes-p-bx-080816 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:17 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:17 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:18 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:18 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:18 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-6-x-6-standard-boxes-p-bx-080606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-9-x-3-flat-boxes-p-bx-120903 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/26-x-15-x-5-flat-boxes-p-bx-261505 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:19 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-12-x-2-flat-boxes-p-bx-121202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:20 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:20 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/36-x-8-x-8-long-boxes-p-bx-360808 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:20 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/international-safe-handling-labels/keep-refrigerated-shipping-lables-international-3x4-p-ipm315 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shockwatch-labels/shockwatch-2-15g-label-p-si-47000k already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:21 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/24-x-20-x-46-wardrobe-moving-boxes-p-bx-242046w already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-48-x-48-x-72-clear-plastic-pallet-covers-p-pc2484872 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:21 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/44-x-12-x-12-standard-boxes-p-bx-441212 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/28-x-5-x-24-standard-boxes-p-bx-280524fol already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:23 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:23 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:24 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/fragile-labels/fragile-handle-with-care-5x3-warning-labels-p-scl536 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:24 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:25 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/10x10-4mil-ziplock-bags-p-cz41010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-48-x-42-x-66-clear-plastic-pallet-covers-p-pc3484266 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/60-x-60-2-mil-pallet-top-sheets-p-pts26060 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:26 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/24-x-24-x-24-deluxe-moving-boxes-p-bx-242424d already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:26 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:27 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/poly-tubing/18in-6-mil-black-tubing-on-roll-p-ptb618 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/poly-tubing/12in-6-mil-black-tubing-on-roll-p-ptb612 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-10-x-4-flat-boxes-p-bx-201004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-8-x-17-tall-boxes-p-bx-080817 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:28 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tiltwatch/tiltwatch-xtr-p-tw-24101 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:28 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:29 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:29 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/shrink-wrap/bundling-film/22in-x-4200ft-2-mil-shrink-bundling-film-p-ps2022 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/6-x-6-x-18-tall-boxes-p-bx-060618 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:30 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:30 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/heavy-duty/16-x-12-x-12-heavy-duty-boxes-p-bx-161212hddw already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-6-x-6-long-boxes-p-bx-120606 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-8-x-6-standard-boxes-p-bx-120806 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/6-x-6-x-4-standard-boxes-p-bx-060604 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-18-x-8-standard-boxes-p-bx-181808 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/6-x-6-x-14-tall-boxes-p-bx-060614 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:31 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/international-safe-handling-labels/3x4-do-not-stack-in-transit-labels-p-ipm2150 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-12-x-6-flat-boxes-p-bx-201206 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:31 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/newsprint-paper/18-x-1440-newsprint-roll-p-npr-1890 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:31 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-9-x-4-white-corrugated-boxes-p-bx-120904w already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:32 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:32 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-48-x-42-x-48-clear-plastic-pallet-covers-p-pc2484248 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:33 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/10-x-6-x-57-moving-ski-boxes-p-bx-100657skib already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:33 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:33 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:33 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/24-x-16-x-18-standard-boxes-p-bx-241618 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/4-x-4-x-4-cube-boxes-p-bx-040404 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/tan-2.1-mil-2-in-x-110-yds-pvc-natural-rubber-p-tprpvc2102110-tan already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/twisted-handle-shopping-bags/8-x-4-x-10-navy-blue-twisted-paper-handle-shoppers-p-mb-thp-8-navy already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:34 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/20-x-8-x-8-long-boxes-p-bx-200808 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:35 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:36 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:36 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/6x9-4mil-ziplock-bags-p-cz40609 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-48-x-42-x-48-clear-plastic-pallet-covers-p-pc3484248 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/22-x-12-x-10-standard-boxes-p-bx-221210 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:37 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:37 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/moving-boxes/18-x-18-x-16-deluxe-moving-boxes-p-bx-181812d already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:38 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:38 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:39 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:39 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:39 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2mil-48-x-46-x-72-black-plastic-pallet-covers-p-pcb2484672 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/18-x-10-x-6-standard-boxes-p-bx-181006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/7-x-7-x-4-standard-boxes-p-bx-070704 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-9-x-5-standard-boxes-p-bx-120905 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/42-x-32-x-54-4-mil-clear-plastic-pallet-covers-p-pc4423254 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/9-x-5-x-5-standard-boxes-p-bx-090505 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8.75-x-4.375-x-9.5-standard-boxes-p-bx-080409 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/13-x-13-x-3-flat-boxes-p-bx-131303 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/20x20-4mil-ziplock-bags-p-cz42020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/heavy-duty/48-x-36-x-40-heavy-duty-boxes-p-bx-484036hddw already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:41 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:42 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/16-x-14-x-14-standard-boxes-p-bx-161414 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/8-x-8-x-40-tall-boxes-p-bx-080840 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/12-x-10-x-4-flat-boxes-p-bx-121004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/20x24-4mil-ziplock-bags-p-cz42024 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:43 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:44 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:44 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/3-mil-72-x-42-x-54-clear-plastic-pallet-covers-p-pc3724254 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:44 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:45 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2mil-50x49x75-clear-plastic-pallet-covers-p-pc997520 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/16x20-4mil-ziplock-bags-p-cz41620 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/corrugated-cardboard-boxes/200-lb-standard/22-x-10-x-6-flat-boxes-p-bx-221006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:46 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:46 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/2-mil-46-x-36-x-65-clear-plastic-pallet-covers-p-pc2463665 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/1-mil-48-x-40-x-100-clear-plastic-pallet-covers-p-pc14840100 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/48-x-48-1.5-mil-pallet-top-sheets-p-pts154848 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:47 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:48 [scrapy.core.scraper] (PID: 131) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/1.5mil-42x26x63-clear-plastic-pallet-covers-p-pc686315 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/ziplock-bags/4-mil-clearzip/2x2-4mil-ziplock-bags-p-cz40202 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/oxidizer-dot-labels-p-hml410 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/labels/dot-hazmat-labels/flammable-liquid-dot-labels-p-hml406 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/tape/packing-tape/carton-sealing/acrylic-2-in-x-55-yds-2.6-mil-carton-sealing-tape-p-tl-t901291 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 131) WARNING: Request https://www.interplas.com/pallet-covers/36-x-28-x-96-4-mil-clear-plastic-pallet-covers-p-pc4362896 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:50 [scrapy.core.engine] (PID: 131) INFO: Closing spider (finished) 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] ------------------------------ MONITORS ------------------------------ 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] Extracted Items Monitor/test_stat_monitor... FAIL 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] Item Validation Monitor/test_stat_monitor... SKIPPED (Unable to find 'spidermon/validation/fields/errors' in job stats.) 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] Error Count Monitor/test_stat_monitor... FAIL 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] Warning Count Monitor/test_stat_monitor... OK 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] Finish Reason Monitor/Should have the expected finished reason(s)... OK 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] Unwanted HTTP codes monitor/Should not hit the limit of unwanted http status... OK 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] Field Coverage Monitor/test_check_if_field_coverage_rules_are_met... OK 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] Retry Count monitor/Should not hit the limit of requests that reached the maximum retry amount... OK 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] Downloader Exceptions monitor/test_stat_monitor... FAIL 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] Successful Requests monitor/Should have at least the minimum number of successful requests... OK 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] Total Requests monitor/Should not hit the total limit of requests... OK 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] ---------------------------------------------------------------------- 2025-11-01 13:20:50 [international_plastics] (PID: 131) ERROR: [Spidermon] ====================================================================== FAIL: Extracted Items Monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 184, in test_stat_monitor assertion_method( AssertionError: Expecting 'item_scraped_count' to be '>=' to '7000.0'. Current value: '6972' 2025-11-01 13:20:50 [international_plastics] (PID: 131) ERROR: [Spidermon] ====================================================================== FAIL: Error Count Monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 184, in test_stat_monitor assertion_method( AssertionError: Expecting 'log_count/ERROR' to be '<=' to '10.0'. Current value: '626' 2025-11-01 13:20:50 [international_plastics] (PID: 131) ERROR: [Spidermon] ====================================================================== FAIL: Downloader Exceptions monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 184, in test_stat_monitor assertion_method( AssertionError: Expecting 'downloader/exception_count' to be '<=' to '100.0'. Current value: '625' 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] 11 monitors in 0.004s 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] FAILED (failures=3, skipped=1) 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] -------------------------- FINISHED ACTIONS -------------------------- 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] ---------------------------------------------------------------------- 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] 0 actions in 0.000s 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] OK 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] --------------------------- PASSED ACTIONS --------------------------- 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] ---------------------------------------------------------------------- 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] 0 actions in 0.000s 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] OK 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] --------------------------- FAILED ACTIONS --------------------------- 2025-11-01 13:20:50 [spidermon.contrib.actions.slack] (PID: 131) WARNING: bot cannot finder user in slack org member list - default icon url used 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] CustomTemplateSendSlackMessageSpiderFinished... OK 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] ---------------------------------------------------------------------- 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] 1 action in 0.427s 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: [Spidermon] OK 2025-11-01 13:20:50 [international_plastics] (PID: 131) INFO: 1 URLs returned ProductNotFound. 2025-11-01 13:20:51 [international_plastics] (PID: 131) INFO: URL https://www.interplas.com/trash-bags/green-biodegradable-garbage-bags-p-cl-big-4761 is already flagged as 'DISABLED_NOT_FOUND' in the catalog_urls table. 2025-11-01 13:20:51 [international_plastics] (PID: 131) WARNING: 1 URLs were not found in the `catalog_urls` table. 2025-11-01 13:20:51 [international_plastics] (PID: 131) INFO: URLs not found and not flagged: {'https://www.interplas.com/trash-bags/green-biodegradable-garbage-bags-p-cl-big-4761'} 2025-11-01 13:20:51 [international_plastics] (PID: 131) INFO: Finished processing 'not found' URLs in the `catalog_urls` table. 2025-11-01 13:20:52 [scrapy.extensions.feedexport] (PID: 131) INFO: Stored bq feed (972 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:7 2025-11-01 13:20:52 [scrapy.statscollectors] (PID: 131) INFO: Dumping Scrapy stats: {'HeadersSpooferDownloaderMiddleware/spoofed': 7680, 'NotFoundHandlerSpiderMiddleware/ProductNotFound': 1, 'NotFoundHandlerSpiderMiddleware/not_found/404_response': 1, 'NotFoundHandlerSpiderMiddleware/not_found/ignored': 1, 'NotFoundHandlerSpiderMiddleware/not_found/retrieved': 1, 'big_query/url': 7598, 'downloader/exception_count': 625, 'downloader/exception_type_count/builtins.ValueError': 625, 'downloader/request_bytes': 14175154, 'downloader/request_count': 7680, 'downloader/request_method_count/GET': 7680, 'downloader/response_bytes': 192255371, 'downloader/response_count': 7680, 'downloader/response_status_count/200': 6972, 'downloader/response_status_count/301': 707, 'downloader/response_status_count/404': 1, 'elapsed_time_seconds': 336.775474, 'feedexport/success_count/BigQueryFeedStorage': 7, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2025, 11, 1, 13, 20, 50, 1616, tzinfo=datetime.timezone.utc), 'httpcompression/response_bytes': 1124320250, 'httpcompression/response_count': 6972, 'item_scraped_count': 6972, 'items_per_minute': None, 'log_count/ERROR': 629, 'log_count/INFO': 58, 'log_count/WARNING': 675, 'memusage/max': 257363968, 'memusage/startup': 126349312, 'poet/injector/catalog_extraction.pages.international_plastics.InternationalPlasticsProductsPageObject': 6973, 'product_status/discontinued': 1373, 'product_status/not_available_online': 42, 'proxy_manager/ignored/proxy_defined': 707, 'proxy_manager/processed': 7598, 'response_received_count': 6973, 'responses_per_minute': None, 'scheduler/dequeued': 8305, 'scheduler/dequeued/memory': 8305, 'scheduler/enqueued': 8305, 'scheduler/enqueued/memory': 8305, 'spider_exceptions/ProductNotFound': 1, 'spidermon/validation/fields': 146412, 'spidermon/validation/items': 6972, 'spidermon/validation/validators': 1, 'spidermon/validation/validators/item/jsonschema': True, 'spidermon_field_coverage/dict/brand': 1.0, 'spidermon_field_coverage/dict/categories': 1.0, 'spidermon_field_coverage/dict/countryOfOrigin': 1.0, 'spidermon_field_coverage/dict/description': 1.0, 'spidermon_field_coverage/dict/imageUrl': 1.0, 'spidermon_field_coverage/dict/inStock': 1.0, 'spidermon_field_coverage/dict/isFreeShipping': 1.0, 'spidermon_field_coverage/dict/leadTime': 1.0, 'spidermon_field_coverage/dict/manufacturer': 1.0, 'spidermon_field_coverage/dict/manufacturerSku': 1.0, 'spidermon_field_coverage/dict/name': 1.0, 'spidermon_field_coverage/dict/packagingIncrement': 1.0, 'spidermon_field_coverage/dict/prices': 1.0, 'spidermon_field_coverage/dict/productStatus': 1.0, 'spidermon_field_coverage/dict/relatedSkus': 1.0, 'spidermon_field_coverage/dict/specifications': 1.0, 'spidermon_field_coverage/dict/supplier': 1.0, 'spidermon_field_coverage/dict/supplierSku': 1.0, 'spidermon_field_coverage/dict/uom': 1.0, 'spidermon_field_coverage/dict/url': 1.0, 'spidermon_field_coverage/dict/weight': 1.0, 'spidermon_item_scraped_count': 6972, 'spidermon_item_scraped_count/dict': 6972, 'spidermon_item_scraped_count/dict/brand': 6972, 'spidermon_item_scraped_count/dict/categories': 6972, 'spidermon_item_scraped_count/dict/countryOfOrigin': 6972, 'spidermon_item_scraped_count/dict/description': 6972, 'spidermon_item_scraped_count/dict/imageUrl': 6972, 'spidermon_item_scraped_count/dict/inStock': 6972, 'spidermon_item_scraped_count/dict/isFreeShipping': 6972, 'spidermon_item_scraped_count/dict/leadTime': 6972, 'spidermon_item_scraped_count/dict/manufacturer': 6972, 'spidermon_item_scraped_count/dict/manufacturerSku': 6972, 'spidermon_item_scraped_count/dict/name': 6972, 'spidermon_item_scraped_count/dict/packagingIncrement': 6972, 'spidermon_item_scraped_count/dict/prices': 6972, 'spidermon_item_scraped_count/dict/productStatus': 6972, 'spidermon_item_scraped_count/dict/relatedSkus': 6972, 'spidermon_item_scraped_count/dict/specifications': 6972, 'spidermon_item_scraped_count/dict/supplier': 6972, 'spidermon_item_scraped_count/dict/supplierSku': 6972, 'spidermon_item_scraped_count/dict/uom': 6972, 'spidermon_item_scraped_count/dict/url': 6972, 'spidermon_item_scraped_count/dict/weight': 6972, 'start_requests/big_query': 7598, 'start_time': datetime.datetime(2025, 11, 1, 13, 15, 13, 226142, tzinfo=datetime.timezone.utc)} 2025-11-01 13:20:52 [scrapy.core.engine] (PID: 131) INFO: Spider closed (finished)