2025-11-01 09:15:07 [scrapy.utils.log] (PID: 101) INFO: Scrapy 2.12.0 started (bot: catalog_extraction) 2025-11-01 09:15:07 [scrapy.utils.log] (PID: 101) INFO: Versions: lxml 5.3.1.0, libxml2 2.12.9, cssselect 1.3.0, parsel 1.10.0, w3lib 2.3.1, Twisted 24.11.0, Python 3.11.13 (main, Jun 10 2025, 23:54:42) [GCC 12.2.0], pyOpenSSL 25.0.0 (OpenSSL 3.4.1 11 Feb 2025), cryptography 44.0.2, Platform Linux-6.9.12-x86_64-with-glibc2.36 2025-11-01 09:15:07 [smith_corona] (PID: 101) INFO: Starting extraction spider smith_corona... 2025-11-01 09:15:07 [scrapy.addons] (PID: 101) INFO: Enabled addons: [] 2025-11-01 09:15:07 [py.warnings] (PID: 101) WARNING: /usr/local/lib/python3.11/site-packages/scrapy/utils/request.py:120: ScrapyDeprecationWarning: 'REQUEST_FINGERPRINTER_IMPLEMENTATION' is a deprecated setting. It will be removed in a future version of Scrapy. return cls(crawler) 2025-11-01 09:15:07 [scrapy.extensions.telnet] (PID: 101) INFO: Telnet Password: 6ab6f92e705e224f 2025-11-01 09:15:07 [py.warnings] (PID: 101) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 09:15:07 [scrapy.middleware] (PID: 101) INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.closespider.CloseSpider', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats', 'spidermon.contrib.scrapy.extensions.Spidermon'] 2025-11-01 09:15:07 [scrapy.crawler] (PID: 101) INFO: Overridden settings: {'BOT_NAME': 'catalog_extraction', 'CONCURRENT_ITEMS': 250, 'CONCURRENT_REQUESTS': 24, 'DOWNLOAD_DELAY': 1.25, 'FEED_EXPORT_ENCODING': 'utf-8', 'LOG_FILE': '/var/lib/scrapyd/logs/catalog_extraction/smith_corona/3eaf0dbcb70311f0aeb54200a9fe0102.log', 'LOG_FORMAT': '%(asctime)s [%(name)s] (PID: %(process)d) %(levelname)s: ' '%(message)s', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'catalog_extraction.spiders', 'REQUEST_FINGERPRINTER_CLASS': 'scrapy_poet.ScrapyPoetRequestFingerprinter', 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7', 'RETRY_HTTP_CODES': [500, 502, 503, 504, 522, 524, 408, 429, 403], 'RETRY_TIMES': 5, 'SPIDER_MODULES': ['catalog_extraction.spiders'], 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor', 'USER_AGENT': None} 2025-11-01 09:15:08 [scrapy_poet.injection] (PID: 101) INFO: Loading providers: [, , , , , , ] 2025-11-01 09:15:08 [scrapy.middleware] (PID: 101) INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scraping_utils.middlewares.downloaders.ProxyManagerDownloaderMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scraping_utils.middlewares.downloaders.HeadersSpooferDownloaderMiddleware', 'scrapy_poet.InjectionMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy_poet.DownloaderStatsMiddleware'] 2025-11-01 09:15:08 [NotFoundHandlerSpiderMiddleware] (PID: 101) INFO: NotFoundHandlerSpiderMiddleware running on PRODUCTION environment. 2025-11-01 09:15:08 [scrapy.middleware] (PID: 101) INFO: Enabled spider middlewares: ['catalog_extraction.middlewares.NotFoundHandlerSpiderMiddleware', 'catalog_extraction.middlewares.FixtureSavingMiddleware', 'scrapy_poet.RetryMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2025-11-01 09:15:08 [scrapy.middleware] (PID: 101) INFO: Enabled item pipelines: ['catalog_extraction.pipelines.DuplicatedSKUsFilterPipeline', 'catalog_extraction.pipelines.DiscontinuedProductsAdjustmentPipeline', 'catalog_extraction.pipelines.PriceRoundingPipeline', 'scraping_utils.pipelines.AttachSupplierPipeline', 'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline'] 2025-11-01 09:15:08 [scrapy.core.engine] (PID: 101) INFO: Spider opened 2025-11-01 09:15:08 [scrapy.extensions.closespider] (PID: 101) INFO: Spider will stop when no items are produced after 1800 seconds. 2025-11-01 09:15:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2025-11-01 09:15:08 [scrapy.extensions.telnet] (PID: 101) INFO: Telnet console listening on 127.0.0.1:6025 2025-11-01 09:15:10 [ProxyManagerDownloaderMiddleware] (PID: 101) INFO: Using brd-customer-hl_13cda1e4-zone-sharedpool_datacenter_proxy as the default proxy for ProxyManagerDownloaderMiddleware. 2025-11-01 09:15:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:15:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:15:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:15:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:15:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG4025APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG4070APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG40130APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S927BB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-RED-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4080AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-CHARTREUSE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 36 pages (at 36 pages/min), scraped 12 items (at 12 items/min) 2025-11-01 09:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-875-x-0-875-sheet-labels-blockout-81.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 55 pages (at 19 pages/min), scraped 24 items (at 12 items/min) 2025-11-01 09:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:17:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:17:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-2-sheet-labels-blockout-87.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-875-x-0-875-sheet-labels-blockout-81.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-blockout-106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:17:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-blockout-137.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 55 pages (at 0 pages/min), scraped 24 items (at 0 items/min) 2025-11-01 09:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-2-sheet-labels-blockout-87.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-875-x-0-875-sheet-labels-blockout-81.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-blockout-106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-5-5-sheet-labels-blockout-205.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-2-sheet-labels-blockout-87.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-875-x-0-875-sheet-labels-blockout-81.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-blockout-106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 55 pages (at 0 pages/min), scraped 24 items (at 0 items/min) 2025-11-01 09:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-5-5-sheet-labels-blockout-205.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-65-x-4-65-sheet-labels-blockout-495.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT6040A-TOP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-2-sheet-labels-blockout-87.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:39 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:19:39 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-polypropylene.html>: HTTP status code is not handled or not allowed 2025-11-01 09:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-blockout-106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:45 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:19:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:45 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-grape.html>: HTTP status code is not handled or not allowed 2025-11-01 09:19:52 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 09:19:52 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core.html>: HTTP status code is not handled or not allowed 2025-11-01 09:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:05 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 09:20:05 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-red.html>: HTTP status code is not handled or not allowed 2025-11-01 09:20:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 78 pages (at 23 pages/min), scraped 30 items (at 6 items/min) 2025-11-01 09:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:20:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:20:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:20:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4065AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4065AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4090AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:21:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 118 pages (at 40 pages/min), scraped 47 items (at 17 items/min) 2025-11-01 09:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:21:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:17 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:21:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:21:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:21:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:21:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 157 pages (at 39 pages/min), scraped 67 items (at 20 items/min) 2025-11-01 09:22:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:22:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 197 pages (at 40 pages/min), scraped 89 items (at 22 items/min) 2025-11-01 09:23:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:23:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 238 pages (at 41 pages/min), scraped 109 items (at 20 items/min) 2025-11-01 09:24:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:24:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:24:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:24:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:24:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:24:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 279 pages (at 41 pages/min), scraped 125 items (at 16 items/min) 2025-11-01 09:25:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:25:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:25:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:25:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:25:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:25:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:25:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 318 pages (at 39 pages/min), scraped 143 items (at 18 items/min) 2025-11-01 09:26:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:26:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:26:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:26:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 354 pages (at 36 pages/min), scraped 162 items (at 19 items/min) 2025-11-01 09:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:27:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 394 pages (at 40 pages/min), scraped 183 items (at 21 items/min) 2025-11-01 09:28:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:28:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:29:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 431 pages (at 37 pages/min), scraped 199 items (at 16 items/min) 2025-11-01 09:29:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:29:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:29:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:29:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:29:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 470 pages (at 39 pages/min), scraped 218 items (at 19 items/min) 2025-11-01 09:30:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:30:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:30:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:30:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 509 pages (at 39 pages/min), scraped 239 items (at 21 items/min) 2025-11-01 09:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:31:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:31:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:31:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:31:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:31:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 551 pages (at 42 pages/min), scraped 258 items (at 19 items/min) 2025-11-01 09:32:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=Z*0327*1476*W*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:32:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-laser-sheet-label.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:32:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-HOTPINK-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=9401 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT2312540A-DYMO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:33:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 587 pages (at 36 pages/min), scraped 277 items (at 19 items/min) 2025-11-01 09:33:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:33:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:33:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-CHARTREUSE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:33:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:34:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:34:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 615 pages (at 28 pages/min), scraped 289 items (at 12 items/min) 2025-11-01 09:34:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3040AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:34:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 636 pages (at 21 pages/min), scraped 300 items (at 11 items/min) 2025-11-01 09:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3040AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:35:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:35:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:35:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3040AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:35:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:56 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:35:56 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-red-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 09:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:35:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 638 pages (at 2 pages/min), scraped 300 items (at 0 items/min) 2025-11-01 09:36:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:36:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-9375-x-0-625-sheet-labels-blockout-42.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3040AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:36:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-direct-thermal-labels-1-core-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-9375-x-0-625-sheet-labels-blockout-42.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3040AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-sheet-labels-blockout-90.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 638 pages (at 0 pages/min), scraped 300 items (at 0 items/min) 2025-11-01 09:37:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-direct-thermal-labels-1-core-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-9375-x-0-625-sheet-labels-blockout-42.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-sheet-labels-blockout-90.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:42 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:37:42 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-red-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 09:37:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:37:44 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:37:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:44 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-chartreuse-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 09:37:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-direct-thermal-labels-1-core-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:48 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:37:48 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 09:37:48 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 09:37:48 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 09:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:53 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:37:53 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-GREEN>: HTTP status code is not handled or not allowed 2025-11-01 09:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:37:56 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:37:56 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 09:37:57 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:37:57 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-GRAY>: HTTP status code is not handled or not allowed 2025-11-01 09:37:58 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:37:58 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-PINK>: HTTP status code is not handled or not allowed 2025-11-01 09:37:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-9375-x-0-625-sheet-labels-blockout-42.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 649 pages (at 11 pages/min), scraped 301 items (at 1 items/min) 2025-11-01 09:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-sheet-labels-blockout-90.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:13 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:38:13 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-grape.html>: HTTP status code is not handled or not allowed 2025-11-01 09:38:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:17 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:38:17 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-FLRED>: HTTP status code is not handled or not allowed 2025-11-01 09:38:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-blockout-133.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:38:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:38:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:26 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:38:27 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-green.html>: HTTP status code is not handled or not allowed 2025-11-01 09:38:27 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:38:28 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-FLORANGE>: HTTP status code is not handled or not allowed 2025-11-01 09:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-sheet-labels-blockout-204.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:38:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:36 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:38:36 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-fluorescent-red.html>: HTTP status code is not handled or not allowed 2025-11-01 09:38:37 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:38:38 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLUE-XL>: HTTP status code is not handled or not allowed 2025-11-01 09:38:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:38:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-9375-x-0-625-sheet-labels-blockout-42.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:38:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-sheet-labels-blockout-90.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-9375-x-0-625-sheet-labels-white-matte-litho-42.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:50 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:38:50 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-PURPLE>: HTTP status code is not handled or not allowed 2025-11-01 09:38:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-blockout-133.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:38:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:38:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:39:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-sheet-labels-white-matte-litho-130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-3-5-sheet-labels-white-matte-litho-200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-sheet-labels-blockout-204.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:07 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:39:07 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-dark-blue-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 09:39:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 657 pages (at 8 pages/min), scraped 301 items (at 0 items/min) 2025-11-01 09:39:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-sheet-labels-white-matte-litho-201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-5-x-2-sheet-labels-white-matte-litho-207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:19 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:39:19 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/1-9375-x-0-625-sheet-labels-blockout-42.html>: HTTP status code is not handled or not allowed 2025-11-01 09:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-sheet-labels-blockout-90.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-9375-x-0-625-sheet-labels-white-matte-litho-42.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-96-x-0-46-sheet-labels-white-matte-litho-570.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-blockout-133.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:39:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:39:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-sheet-labels-white-matte-litho-877.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-sheet-labels-white-matte-litho-130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-3-5-sheet-labels-white-matte-litho-200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-9-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-sheet-labels-white-matte-litho-201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-5-x-2-sheet-labels-white-matte-litho-207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-x-10-sheet-labels-photo-gloss-78.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:39:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:02 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:40:02 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-sheet-labels-blockout-90.html>: HTTP status code is not handled or not allowed 2025-11-01 09:40:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-9375-x-0-625-sheet-labels-white-matte-litho-42.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-96-x-0-46-sheet-labels-white-matte-litho-570.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 659 pages (at 2 pages/min), scraped 301 items (at 0 items/min) 2025-11-01 09:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4065AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT6090AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:40:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4100AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:40:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:40:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 696 pages (at 37 pages/min), scraped 314 items (at 13 items/min) 2025-11-01 09:41:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:41:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-white-premium-high-gloss-355.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-0-75-sheet-labels-white-premium-high-gloss-505.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-656-x-4-656-sheet-labels-clear-matte-97.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-sheet-labels-clear-matte-500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-1-75-sheet-labels-clear-matte-590.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-sheet-labels-clear-matte-877.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 728 pages (at 32 pages/min), scraped 327 items (at 13 items/min) 2025-11-01 09:42:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-crystal-clear-117.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:42:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:43:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 765 pages (at 37 pages/min), scraped 344 items (at 17 items/min) 2025-11-01 09:43:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:43:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:43:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:43:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:43:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 805 pages (at 40 pages/min), scraped 366 items (at 22 items/min) 2025-11-01 09:44:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:44:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:44:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:44:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 845 pages (at 40 pages/min), scraped 385 items (at 19 items/min) 2025-11-01 09:45:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:45:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:45:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 883 pages (at 38 pages/min), scraped 402 items (at 17 items/min) 2025-11-01 09:46:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:46:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:46:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:46:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:46:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 924 pages (at 41 pages/min), scraped 421 items (at 19 items/min) 2025-11-01 09:47:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:47:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:47:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:47:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:48:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:48:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:48:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 961 pages (at 37 pages/min), scraped 439 items (at 18 items/min) 2025-11-01 09:48:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:48:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:48:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:48:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:48:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:48:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:48:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:48:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:48:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:49:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 999 pages (at 38 pages/min), scraped 456 items (at 17 items/min) 2025-11-01 09:49:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:49:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:49:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1040 pages (at 41 pages/min), scraped 472 items (at 16 items/min) 2025-11-01 09:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG225125APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG4065APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG5030APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP*2 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040A-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=9380 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:50:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:51:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:51:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1079 pages (at 39 pages/min), scraped 493 items (at 21 items/min) 2025-11-01 09:51:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S908BB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4060 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:51:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4080 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:51:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4060AP-TOP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1114 pages (at 35 pages/min), scraped 511 items (at 18 items/min) 2025-11-01 09:52:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3100AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSC0005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:52:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-FLGREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-GRAPE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:52:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:52:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:53:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:53:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1145 pages (at 31 pages/min), scraped 526 items (at 15 items/min) 2025-11-01 09:53:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:53:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:53:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-625-x-1-625-sheet-labels-blockout-60.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:53:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:53:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:53:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-625-sheet-labels-blockout-108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1164 pages (at 19 pages/min), scraped 535 items (at 9 items/min) 2025-11-01 09:54:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:54:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:54:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:54:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-625-sheet-labels-blockout-108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-sheet-labels-blockout-201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-0-75-sheet-labels-blockout-220.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:54:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:00 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:55:00 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-brown.html>: HTTP status code is not handled or not allowed 2025-11-01 09:55:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1165 pages (at 1 pages/min), scraped 535 items (at 0 items/min) 2025-11-01 09:55:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:10 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:55:10 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 09:55:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:55:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:55:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-625-sheet-labels-blockout-108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-sheet-labels-blockout-201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-0-75-sheet-labels-blockout-220.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-1-5-sheet-labels-blockout-390.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-1-5-sheet-labels-blockout-400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-625-sheet-labels-blockout-108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:55:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:56:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-sheet-labels-blockout-201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:05 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:56:05 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-green.html>: HTTP status code is not handled or not allowed 2025-11-01 09:56:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:56:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1167 pages (at 2 pages/min), scraped 535 items (at 0 items/min) 2025-11-01 09:56:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:56:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-1-5-sheet-labels-blockout-400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:31 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 09:56:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:31 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blue-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 09:56:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:56:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-625-sheet-labels-blockout-108.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-sheet-labels-blockout-201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-75-x-2-5-sheet-labels-blockout-867.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-2-75-sheet-labels-blockout-890.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:46 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:56:46 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-red.html>: HTTP status code is not handled or not allowed 2025-11-01 09:56:48 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:56:48 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-red.html>: HTTP status code is not handled or not allowed 2025-11-01 09:56:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:54 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:56:54 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-pink-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 09:56:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:56:56 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:56:56 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-FLRED>: HTTP status code is not handled or not allowed 2025-11-01 09:56:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:56:58 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:56:59 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-ORANGE>: HTTP status code is not handled or not allowed 2025-11-01 09:57:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-1-5-sheet-labels-blockout-400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-5-x-4-125-sheet-labels-white-matte-litho-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:05 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:57:05 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-purple.html>: HTTP status code is not handled or not allowed 2025-11-01 09:57:07 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:57:07 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AF-XL>: HTTP status code is not handled or not allowed 2025-11-01 09:57:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1176 pages (at 9 pages/min), scraped 535 items (at 0 items/min) 2025-11-01 09:57:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:57:09 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:57:09 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-625-sheet-labels-blockout-108.html>: HTTP status code is not handled or not allowed 2025-11-01 09:57:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-sheet-labels-blockout-201.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:13 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:57:13 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-dark-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 09:57:15 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:57:15 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLOCKOUT-XL>: HTTP status code is not handled or not allowed 2025-11-01 09:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-75-x-2-5-sheet-labels-blockout-867.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:18 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:57:18 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-FLGREEN>: HTTP status code is not handled or not allowed 2025-11-01 09:57:19 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:57:19 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-FLGREEN>: HTTP status code is not handled or not allowed 2025-11-01 09:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-2-75-sheet-labels-blockout-890.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-white-matte-litho-70.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-sheet-labels-white-matte-litho-91.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:26 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:57:27 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-GRAY>: HTTP status code is not handled or not allowed 2025-11-01 09:57:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-sheet-labels-white-matte-litho-120.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-4-75-sheet-labels-white-matte-litho-203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:33 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:57:33 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 09:57:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-sheet-labels-white-matte-litho-280.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-1-5-sheet-labels-blockout-400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:57:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-white-matte-litho-421.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-25-x-3-sheet-labels-white-matte-litho-460.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:49 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:57:49 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-3-sheet-labels-blockout-201.html>: HTTP status code is not handled or not allowed 2025-11-01 09:57:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-75-x-2-5-sheet-labels-blockout-867.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:57:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-2-75-sheet-labels-blockout-890.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-white-matte-litho-70.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-sheet-labels-white-matte-litho-91.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-75-x-2-75-sheet-labels-white-matte-litho-822.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:05 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:58:05 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 09:58:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-sheet-labels-white-matte-litho-120.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1185 pages (at 9 pages/min), scraped 535 items (at 0 items/min) 2025-11-01 09:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-4-75-sheet-labels-white-matte-litho-203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-33-sheet-labels-white-matte-litho-824.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-sheet-labels-white-matte-litho-280.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-1-5-sheet-labels-blockout-400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-5-375-sheet-labels-white-matte-litho-872.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:58:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-white-matte-litho-421.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-25-x-3-sheet-labels-white-matte-litho-460.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:58:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-1-sheet-labels-photo-gloss-16.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-75-x-2-5-sheet-labels-blockout-867.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:58:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-2-75-sheet-labels-blockout-890.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-white-matte-litho-70.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:58:38 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:58:39 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-pink-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 09:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-75-x-2-75-sheet-labels-white-matte-litho-822.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-11-sheet-labels-photo-gloss-93.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-sheet-labels-white-matte-litho-120.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-4-75-sheet-labels-white-matte-litho-203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-33-sheet-labels-white-matte-litho-824.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-sheet-labels-white-matte-litho-280.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:48 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 09:58:48 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-25-x-1-5-sheet-labels-blockout-400.html>: HTTP status code is not handled or not allowed 2025-11-01 09:58:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-5-375-sheet-labels-white-matte-litho-872.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-x-0-375-sheet-labels-photo-gloss-112.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-white-matte-litho-421.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:58:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-25-x-3-sheet-labels-white-matte-litho-460.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1190 pages (at 5 pages/min), scraped 535 items (at 0 items/min) 2025-11-01 09:59:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 09:59:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 09:59:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:00:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:00:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1231 pages (at 41 pages/min), scraped 555 items (at 20 items/min) 2025-11-01 10:00:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:00:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:00:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1272 pages (at 41 pages/min), scraped 571 items (at 16 items/min) 2025-11-01 10:01:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:01:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:02:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:02:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:02:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:02:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:02:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:02:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1312 pages (at 40 pages/min), scraped 586 items (at 15 items/min) 2025-11-01 10:02:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:02:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:02:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:02:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:02:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:02:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:02:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:02:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:02:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:02:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:03:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1348 pages (at 36 pages/min), scraped 606 items (at 20 items/min) 2025-11-01 10:03:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:03:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:03:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:04:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1388 pages (at 40 pages/min), scraped 628 items (at 22 items/min) 2025-11-01 10:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:04:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1429 pages (at 41 pages/min), scraped 649 items (at 21 items/min) 2025-11-01 10:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:05:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:05:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:05:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:06:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1468 pages (at 39 pages/min), scraped 665 items (at 16 items/min) 2025-11-01 10:06:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:06:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:06:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:06:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:07:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:07:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1507 pages (at 39 pages/min), scraped 683 items (at 18 items/min) 2025-11-01 10:07:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:07:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:07:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1547 pages (at 40 pages/min), scraped 703 items (at 20 items/min) 2025-11-01 10:08:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:08:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:08:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:08:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:09:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1586 pages (at 39 pages/min), scraped 725 items (at 22 items/min) 2025-11-01 10:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:09:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:09:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:09:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:10:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG2060APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG3010APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1625 pages (at 39 pages/min), scraped 742 items (at 17 items/min) 2025-11-01 10:10:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S385WX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:10:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GREEN-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-ORANGE-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1663 pages (at 38 pages/min), scraped 760 items (at 18 items/min) 2025-11-01 10:11:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:11:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-non-perforated-film-liner.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:11:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S385WX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GREEN-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-ORANGE-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-non-perforated-film-liner.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:11:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-fanfold-labels-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:12:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1666 pages (at 3 pages/min), scraped 762 items (at 2 items/min) 2025-11-01 10:12:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GREEN-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-ORANGE-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-non-perforated-film-liner.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-fanfold-labels-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GREEN-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:12:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:12:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1666 pages (at 0 pages/min), scraped 762 items (at 0 items/min) 2025-11-01 10:13:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-non-perforated-film-liner.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-fanfold-labels-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:13:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:13:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GREEN-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:13:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:13:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-fanfold-labels-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GREEN-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:14:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-RED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1673 pages (at 7 pages/min), scraped 765 items (at 3 items/min) 2025-11-01 10:14:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:12 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:14:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:12 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-direct-thermal-labels-top-coated.html>: HTTP status code is not handled or not allowed 2025-11-01 10:14:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:16 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:14:16 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183W>: HTTP status code is not handled or not allowed 2025-11-01 10:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:21 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:14:21 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195W>: HTTP status code is not handled or not allowed 2025-11-01 10:14:22 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:14:22 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490W>: HTTP status code is not handled or not allowed 2025-11-01 10:14:24 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:14:24 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 10:14:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:27 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:14:27 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 10:14:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:38 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:14:38 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871W>: HTTP status code is not handled or not allowed 2025-11-01 10:14:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:41 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:14:41 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-3-core-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 10:14:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-fanfold-labels-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:44 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:14:44 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GREEN-XS>: HTTP status code is not handled or not allowed 2025-11-01 10:14:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-RED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:49 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:14:49 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 10:14:50 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:14:51 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-12-thermal-transfer-fanfold-labels-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 10:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:54 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:14:54 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 10:14:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:14:57 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:14:57 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 10:14:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:14:59 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:14:59 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-GRAY>: HTTP status code is not handled or not allowed 2025-11-01 10:15:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-0-75-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1687 pages (at 14 pages/min), scraped 765 items (at 0 items/min) 2025-11-01 10:15:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-fanfold-labels-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-RED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-0-75-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-fanfold-labels-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-RED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:16:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1689 pages (at 2 pages/min), scraped 766 items (at 1 items/min) 2025-11-01 10:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:17 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-0-75-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-fanfold-labels-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-RED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:16:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1699 pages (at 10 pages/min), scraped 767 items (at 1 items/min) 2025-11-01 10:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:17:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:17:53 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:17:53 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-green.html>: HTTP status code is not handled or not allowed 2025-11-01 10:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1730 pages (at 31 pages/min), scraped 784 items (at 17 items/min) 2025-11-01 10:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-sheet-labels-blockout-120.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-sheet-labels-blockout-195.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-sheet-labels-blockout-199.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-5-sheet-labels-blockout-330.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-372-x-2-5-sheet-labels-blockout-535.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-42-x-0-58-sheet-labels-blockout-675.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-5-sheet-labels-blockout-770.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:35 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:18:35 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-green-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 10:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:18:42 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:18:42 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-fluorescent-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 10:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:18:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-sheet-labels-blockout-120.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:19:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-sheet-labels-blockout-195.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-sheet-labels-blockout-199.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-5-sheet-labels-blockout-330.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1732 pages (at 2 pages/min), scraped 784 items (at 0 items/min) 2025-11-01 10:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-372-x-2-5-sheet-labels-blockout-535.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-42-x-0-58-sheet-labels-blockout-675.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-5-sheet-labels-blockout-770.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-656-x-4-656-sheet-labels-white-matte-litho-35.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-75-x-4-5-sheet-labels-white-matte-litho-47.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-625-x-1-625-sheet-labels-white-matte-litho-60.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-sheet-labels-blockout-195.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-sheet-labels-blockout-199.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-5-sheet-labels-blockout-330.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-372-x-2-5-sheet-labels-blockout-535.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-42-x-0-58-sheet-labels-blockout-675.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-5-sheet-labels-blockout-770.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:19:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:19:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-75-x-4-5-sheet-labels-white-matte-litho-47.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1732 pages (at 0 pages/min), scraped 784 items (at 0 items/min) 2025-11-01 10:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:20:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-625-x-1-625-sheet-labels-white-matte-litho-60.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-sheet-labels-blockout-195.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-sheet-labels-blockout-199.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:20:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-5-sheet-labels-blockout-330.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-372-x-2-5-sheet-labels-blockout-535.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:20:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-5-sheet-labels-blockout-770.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-9-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4090AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4065AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:20:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:21:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1750 pages (at 18 pages/min), scraped 786 items (at 2 items/min) 2025-11-01 10:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4090AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:14 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:21:14 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 10:21:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:21:17 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:21:17 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-red-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 10:21:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:21:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:22:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1789 pages (at 39 pages/min), scraped 805 items (at 19 items/min) 2025-11-01 10:22:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:22:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1827 pages (at 38 pages/min), scraped 827 items (at 22 items/min) 2025-11-01 10:23:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:23:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:23:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:23:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:23:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:23:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:24:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1864 pages (at 37 pages/min), scraped 843 items (at 16 items/min) 2025-11-01 10:24:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:24:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:24:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:24:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:24:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:24:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:24:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1903 pages (at 39 pages/min), scraped 859 items (at 16 items/min) 2025-11-01 10:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:25:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-125-sheet-labels-pastel-purple-855.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:25:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:26:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-625-x-1-625-sheet-labels-pastel-blue-60.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1938 pages (at 35 pages/min), scraped 874 items (at 15 items/min) 2025-11-01 10:26:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5625-x-1-75-sheet-labels-pastel-blue-440.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 1975 pages (at 37 pages/min), scraped 892 items (at 18 items/min) 2025-11-01 10:27:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:27:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:27:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:27:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:27:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2013 pages (at 38 pages/min), scraped 915 items (at 23 items/min) 2025-11-01 10:28:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:28:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:28:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2054 pages (at 41 pages/min), scraped 934 items (at 19 items/min) 2025-11-01 10:29:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:29:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2092 pages (at 38 pages/min), scraped 951 items (at 17 items/min) 2025-11-01 10:30:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:30:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:30:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2131 pages (at 39 pages/min), scraped 970 items (at 19 items/min) 2025-11-01 10:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:31:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=D*0433*1182*X*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:31:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4065AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-B already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FFT4060-Y already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTF4060AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:31:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:32:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:32:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2169 pages (at 38 pages/min), scraped 989 items (at 19 items/min) 2025-11-01 10:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S912BB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSI0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF4020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4050AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:50 [py.warnings] (PID: 101) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 10:32:51 [scrapy.extensions.feedexport] (PID: 101) INFO: Stored bq feed (1000 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:1 2025-11-01 10:32:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:32:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-SENSOR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2198 pages (at 29 pages/min), scraped 1002 items (at 13 items/min) 2025-11-01 10:33:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSI0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-3-core-poly-freezer.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-direct-thermal-labels-3-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF4020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4050AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-SENSOR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSI0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-3-core-poly-freezer.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-direct-thermal-labels-3-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF4020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:33:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4050AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2198 pages (at 0 pages/min), scraped 1002 items (at 0 items/min) 2025-11-01 10:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-SENSOR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSI0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-3-core-poly-freezer.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-direct-thermal-labels-3-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF4020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4050AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-SENSOR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:34:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:34:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:34:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSI0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-3-core-poly-freezer.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:35:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF4020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2198 pages (at 0 pages/min), scraped 1002 items (at 0 items/min) 2025-11-01 10:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0014 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4050AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:18 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:35:18 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-green.html>: HTTP status code is not handled or not allowed 2025-11-01 10:35:20 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:35:20 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 10:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:27 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:35:27 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-pink-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 10:35:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-SENSOR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSI0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDPF4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:35:49 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:49 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-chartreuse.html>: HTTP status code is not handled or not allowed 2025-11-01 10:35:53 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:35:53 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-orange-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 10:35:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:35:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:35:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:03 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:36:03 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLRED>: HTTP status code is not handled or not allowed 2025-11-01 10:36:04 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:36:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-1-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:04 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-red.html>: HTTP status code is not handled or not allowed 2025-11-01 10:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:36:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2216 pages (at 18 pages/min), scraped 1009 items (at 7 items/min) 2025-11-01 10:36:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4020AP-TOP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-1-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:44 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:36:44 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 10:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-blockout-127.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:36:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4020AP-TOP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:37:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:37:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2223 pages (at 7 pages/min), scraped 1013 items (at 4 items/min) 2025-11-01 10:37:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-1-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-blockout-127.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4020AP-TOP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-1-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:37:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-blockout-127.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2223 pages (at 0 pages/min), scraped 1013 items (at 0 items/min) 2025-11-01 10:38:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-6562-x-2-75-sheet-labels-blockout-475.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4020AP-TOP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:38:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-1-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-blockout-127.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:39 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 10:38:39 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-brown.html>: HTTP status code is not handled or not allowed 2025-11-01 10:38:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:38:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-6562-x-2-75-sheet-labels-blockout-475.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:38:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:59 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:38:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:38:59 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-red.html>: HTTP status code is not handled or not allowed 2025-11-01 10:39:01 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:39:01 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-red.html>: HTTP status code is not handled or not allowed 2025-11-01 10:39:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2232 pages (at 9 pages/min), scraped 1015 items (at 2 items/min) 2025-11-01 10:39:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4020AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:39:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4010AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:39:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:35 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:39:35 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-red-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 10:39:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:40 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:40 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-polypropylene.html>: HTTP status code is not handled or not allowed 2025-11-01 10:39:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:39:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4065AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2275 pages (at 43 pages/min), scraped 1033 items (at 18 items/min) 2025-11-01 10:40:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4090AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:40:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:40:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:40:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:40:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:40:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:40:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:40:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:40:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:41:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2312 pages (at 37 pages/min), scraped 1049 items (at 16 items/min) 2025-11-01 10:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:41:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:41:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:41:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2353 pages (at 41 pages/min), scraped 1067 items (at 18 items/min) 2025-11-01 10:42:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:42:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:42:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:42:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2390 pages (at 37 pages/min), scraped 1085 items (at 18 items/min) 2025-11-01 10:43:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:43:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:43:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:43:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:43:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2427 pages (at 37 pages/min), scraped 1104 items (at 19 items/min) 2025-11-01 10:44:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:44:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:44:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:44:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2469 pages (at 42 pages/min), scraped 1124 items (at 20 items/min) 2025-11-01 10:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:45:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:45:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:45:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:45:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:46:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:46:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2509 pages (at 40 pages/min), scraped 1142 items (at 18 items/min) 2025-11-01 10:46:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:47:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2548 pages (at 39 pages/min), scraped 1160 items (at 18 items/min) 2025-11-01 10:47:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2587 pages (at 39 pages/min), scraped 1178 items (at 18 items/min) 2025-11-01 10:48:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:48:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:49:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2627 pages (at 40 pages/min), scraped 1199 items (at 21 items/min) 2025-11-01 10:49:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:49:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:49:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:49:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:49:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:49:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2667 pages (at 40 pages/min), scraped 1217 items (at 18 items/min) 2025-11-01 10:50:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:50:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG4020APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2706 pages (at 39 pages/min), scraped 1236 items (at 19 items/min) 2025-11-01 10:51:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG3510APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=D*0433*1345*W*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=Z*0433*1476*W*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=Z*0650*1476*W*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:51:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:51:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:51:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S370WX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2744 pages (at 38 pages/min), scraped 1255 items (at 19 items/min) 2025-11-01 10:52:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:52:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S920BB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:52:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSD0004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:52:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:52:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4060-Y already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4040A already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:53:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2772 pages (at 28 pages/min), scraped 1268 items (at 13 items/min) 2025-11-01 10:53:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-PINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:53:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:53:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:53:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4010AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-PINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2799 pages (at 27 pages/min), scraped 1281 items (at 13 items/min) 2025-11-01 10:54:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:54:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:54:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:54:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:54:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2830 pages (at 31 pages/min), scraped 1299 items (at 18 items/min) 2025-11-01 10:55:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4065AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:55:17 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:55:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:17 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-orange-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 10:55:19 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:55:19 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-orange-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 10:55:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:55:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:26 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:55:26 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-grape.html>: HTTP status code is not handled or not allowed 2025-11-01 10:55:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:55:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:49 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:55:49 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 10:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:55:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:56:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:56:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2868 pages (at 38 pages/min), scraped 1313 items (at 14 items/min) 2025-11-01 10:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:56:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:56:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:18 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:56:18 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-dark-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 10:56:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:25 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 10:56:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:26 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 10:56:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:56:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:56:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:56:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:57:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2906 pages (at 38 pages/min), scraped 1327 items (at 14 items/min) 2025-11-01 10:57:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:57:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2944 pages (at 38 pages/min), scraped 1349 items (at 22 items/min) 2025-11-01 10:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:58:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:58:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:58:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:58:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:58:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:58:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:58:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 2987 pages (at 43 pages/min), scraped 1369 items (at 20 items/min) 2025-11-01 10:59:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:59:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 10:59:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 10:59:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3025 pages (at 38 pages/min), scraped 1387 items (at 18 items/min) 2025-11-01 11:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3066 pages (at 41 pages/min), scraped 1404 items (at 17 items/min) 2025-11-01 11:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:01:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:01:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:01:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:01:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3105 pages (at 39 pages/min), scraped 1422 items (at 18 items/min) 2025-11-01 11:02:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:02:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:02:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:02:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:02:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:02:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3142 pages (at 37 pages/min), scraped 1444 items (at 22 items/min) 2025-11-01 11:03:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-11-sheet-labels-brown-kraft-93.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-brown-kraft-117.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-2-sheet-labels-brown-kraft-610.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-875-x-1-0625-sheet-labels-brown-kraft-868.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-375-x-1-25-sheet-labels-brown-kraft-884.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-375-sheet-labels-fluorescent-red-285.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-875-x-3-875-sheet-labels-fluorescent-red-395.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:03:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:03:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-brown-kraft-117.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:03:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:04:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3171 pages (at 29 pages/min), scraped 1460 items (at 16 items/min) 2025-11-01 11:04:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:04:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:04:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:04:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3210 pages (at 39 pages/min), scraped 1476 items (at 16 items/min) 2025-11-01 11:05:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:05:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:05:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-4375-sheet-labels-fluorescent-orange-86.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-5-x-4-5-sheet-labels-fluorescent-yellow-715.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-2-sheet-labels-fluorescent-orange-175.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-656-x-4-656-sheet-labels-fluorescent-orange-97.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:06:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3242 pages (at 32 pages/min), scraped 1491 items (at 15 items/min) 2025-11-01 11:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-sheet-labels-fluorescent-orange-191.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-875-sheet-labels-fluorescent-orange-470.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-1-75-sheet-labels-fluorescent-orange-590.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-fluorescent-orange-720.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-2-sheet-labels-fluorescent-orange-175.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-4375-sheet-labels-fluorescent-orange-86.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-875-x-1-0625-sheet-labels-fluorescent-orange-868.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-sheet-labels-fluorescent-blue-191.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-75-x-0-75-sheet-labels-fluorescent-blue-209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:06:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-1-25-sheet-labels-fluorescent-blue-240.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-875-sheet-labels-fluorescent-orange-470.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-25-x-3-sheet-labels-fluorescent-blue-460.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3256 pages (at 14 pages/min), scraped 1499 items (at 8 items/min) 2025-11-01 11:07:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-fluorescent-orange-720.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-x-1-sheet-labels-fluorescent-blue-630.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-4375-sheet-labels-fluorescent-orange-86.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-75-x-0-75-sheet-labels-fluorescent-blue-209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-4-875-sheet-labels-fluorescent-blue-861.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-625-x-4-625-sheet-labels-fluorescent-blue-875.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-875-sheet-labels-fluorescent-orange-470.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-75-x-9-4-sheet-labels-gold-foil-56.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-fluorescent-orange-720.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-gold-foil-88.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-25-sheet-labels-gold-foil-165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:07:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-4375-sheet-labels-fluorescent-orange-86.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:08:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-5-sheet-labels-gold-foil-325.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-75-x-0-75-sheet-labels-fluorescent-blue-209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3271 pages (at 15 pages/min), scraped 1508 items (at 9 items/min) 2025-11-01 11:08:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-375-x-1-4375-sheet-labels-gold-foil-520.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-875-sheet-labels-fluorescent-orange-470.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-75-x-9-4-sheet-labels-gold-foil-56.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-gold-foil-88.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-x-3-5-sheet-labels-gold-foil-525.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-25-sheet-labels-gold-foil-165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-125-x-5-5-sheet-labels-gold-foil-710.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5625-sheet-labels-gold-foil-785.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-5-sheet-labels-gold-foil-325.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:08:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-7656-x-4-7656-sheet-labels-gold-foil-835.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-silver-foil-70.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-75-x-0-75-sheet-labels-fluorescent-blue-209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:08:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:08:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-875-sheet-labels-fluorescent-orange-470.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-75-x-9-4-sheet-labels-gold-foil-56.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-gold-foil-88.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:09:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-x-3-5-sheet-labels-gold-foil-525.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3286 pages (at 15 pages/min), scraped 1515 items (at 7 items/min) 2025-11-01 11:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5625-sheet-labels-gold-foil-785.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-125-x-5-5-sheet-labels-gold-foil-710.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-875-sheet-labels-silver-foil-470.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-5-sheet-labels-gold-foil-325.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-75-x-0-75-sheet-labels-fluorescent-blue-209.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:09:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-65-x-4-65-sheet-labels-silver-foil-490.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:09:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-75-x-9-4-sheet-labels-gold-foil-56.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG2010APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-inkjet-roll-labels.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-thermal-transfer-labels-3-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-00-x-1476-wax-ribbon.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:09:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:09:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-5-sheet-labels-gold-foil-325.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:01 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:10:01 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/0-75-x-0-75-sheet-labels-fluorescent-blue-209.html>: HTTP status code is not handled or not allowed 2025-11-01 11:10:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=E*0433*245*X*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-65-x-4-65-sheet-labels-silver-foil-490.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3302 pages (at 16 pages/min), scraped 1520 items (at 5 items/min) 2025-11-01 11:10:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-Y already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG4080APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=Z*0400*1476*W*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-FLORG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-Y already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTAG4060AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:10:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=9504 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTP4065AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:10:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3340 pages (at 38 pages/min), scraped 1536 items (at 16 items/min) 2025-11-01 11:11:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:11:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:11:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:11:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S406WX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4020AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:11:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S406WX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3369 pages (at 29 pages/min), scraped 1552 items (at 16 items/min) 2025-11-01 11:12:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-0-75-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4020AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S406WX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:12:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-0-75-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4020AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3373 pages (at 4 pages/min), scraped 1554 items (at 2 items/min) 2025-11-01 11:13:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S406WX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-0-75-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:13:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4020AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S406WX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:13:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:13:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3373 pages (at 0 pages/min), scraped 1554 items (at 0 items/min) 2025-11-01 11:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-0-75-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4020AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S406WX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:14:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:35 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:14:35 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-green-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 11:14:36 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:14:36 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-green-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 11:14:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:51 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 11:14:52 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-brown.html>: HTTP status code is not handled or not allowed 2025-11-01 11:14:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:14:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:15:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:15:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:04 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 11:15:05 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 11:15:05 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:15:05 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 11:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3392 pages (at 19 pages/min), scraped 1562 items (at 8 items/min) 2025-11-01 11:15:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-75-x-4-5-sheet-labels-blockout-47.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-4375-sheet-labels-blockout-85.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:15:50 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:15:50 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 11:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:15:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3399 pages (at 7 pages/min), scraped 1564 items (at 2 items/min) 2025-11-01 11:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-75-x-4-5-sheet-labels-blockout-47.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-4375-sheet-labels-blockout-85.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-2-sheet-labels-blockout-175.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-75-x-4-5-sheet-labels-blockout-47.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-4375-sheet-labels-blockout-85.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:16:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3399 pages (at 0 pages/min), scraped 1564 items (at 0 items/min) 2025-11-01 11:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-2-sheet-labels-blockout-175.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-75-x-4-5-sheet-labels-blockout-47.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:17:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-4375-sheet-labels-blockout-85.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-2-sheet-labels-blockout-175.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:17:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:17:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:18:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:18:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3412 pages (at 13 pages/min), scraped 1568 items (at 4 items/min) 2025-11-01 11:18:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:23 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 11:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:23 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-red.html>: HTTP status code is not handled or not allowed 2025-11-01 11:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:18:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:18:51 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 11:18:51 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-red.html>: HTTP status code is not handled or not allowed 2025-11-01 11:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3449 pages (at 37 pages/min), scraped 1583 items (at 15 items/min) 2025-11-01 11:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT6060AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:19:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3489 pages (at 40 pages/min), scraped 1600 items (at 17 items/min) 2025-11-01 11:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:20:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:20:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:20:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3526 pages (at 37 pages/min), scraped 1617 items (at 17 items/min) 2025-11-01 11:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:21:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:21:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:22:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:22:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3567 pages (at 41 pages/min), scraped 1638 items (at 21 items/min) 2025-11-01 11:22:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:22:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:22:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3606 pages (at 39 pages/min), scraped 1655 items (at 17 items/min) 2025-11-01 11:23:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:23:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:23:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:23:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3645 pages (at 39 pages/min), scraped 1675 items (at 20 items/min) 2025-11-01 11:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:24:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:24:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:24:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:24:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:24:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3682 pages (at 37 pages/min), scraped 1695 items (at 20 items/min) 2025-11-01 11:25:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-3-33-sheet-labels-pastel-orange-863.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-4375-sheet-labels-pastel-tan-85.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-4-15625-sheet-labels-pastel-tan-880.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-x-10-sheet-labels-brown-kraft-78.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:25:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:25:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3717 pages (at 35 pages/min), scraped 1708 items (at 13 items/min) 2025-11-01 11:26:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:26:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:26:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:26:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3755 pages (at 38 pages/min), scraped 1729 items (at 21 items/min) 2025-11-01 11:27:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:27:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:27:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:28:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3797 pages (at 42 pages/min), scraped 1751 items (at 22 items/min) 2025-11-01 11:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:28:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:28:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG4030APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:28:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTPF4020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:29:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3841 pages (at 44 pages/min), scraped 1770 items (at 19 items/min) 2025-11-01 11:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT225125AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT3020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSB0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060AP-B already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:29:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3875 pages (at 34 pages/min), scraped 1786 items (at 16 items/min) 2025-11-01 11:30:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT3020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:30:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT3010AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT3020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:30:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:30:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:30:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-fanfold-labels-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3884 pages (at 9 pages/min), scraped 1791 items (at 5 items/min) 2025-11-01 11:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT3010AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:31:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:31:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT3020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-fanfold-labels-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:31:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT3010AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT3020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3884 pages (at 0 pages/min), scraped 1791 items (at 0 items/min) 2025-11-01 11:32:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-fanfold-labels-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:32:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT3010AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT3020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-fanfold-labels-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:32:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:32:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3896 pages (at 12 pages/min), scraped 1795 items (at 4 items/min) 2025-11-01 11:33:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:33:17 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:33:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4025-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-FLGREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-FLGREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-FLGREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:33:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:34:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3930 pages (at 34 pages/min), scraped 1812 items (at 17 items/min) 2025-11-01 11:34:08 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 11:34:08 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-brown.html>: HTTP status code is not handled or not allowed 2025-11-01 11:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4010AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:34:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-PINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:34:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3965 pages (at 35 pages/min), scraped 1825 items (at 13 items/min) 2025-11-01 11:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:35:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:35:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:35:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 3998 pages (at 33 pages/min), scraped 1845 items (at 20 items/min) 2025-11-01 11:36:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:09 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 11:36:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:09 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 11:36:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:36:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:37:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4037 pages (at 39 pages/min), scraped 1863 items (at 18 items/min) 2025-11-01 11:37:16 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 11:37:16 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-hot-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 11:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:25 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 11:37:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:25 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-gray-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 11:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:37:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:37:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:37:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:37:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4078 pages (at 41 pages/min), scraped 1880 items (at 17 items/min) 2025-11-01 11:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:38:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:38:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4116 pages (at 38 pages/min), scraped 1898 items (at 18 items/min) 2025-11-01 11:39:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:39:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:39:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:40:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:40:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4158 pages (at 42 pages/min), scraped 1919 items (at 21 items/min) 2025-11-01 11:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:40:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:41:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4198 pages (at 40 pages/min), scraped 1938 items (at 19 items/min) 2025-11-01 11:41:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:41:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:42:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:42:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4237 pages (at 39 pages/min), scraped 1956 items (at 18 items/min) 2025-11-01 11:42:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:42:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4278 pages (at 41 pages/min), scraped 1976 items (at 20 items/min) 2025-11-01 11:43:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:43:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4317 pages (at 39 pages/min), scraped 1996 items (at 20 items/min) 2025-11-01 11:44:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:22 [py.warnings] (PID: 101) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 11:44:23 [scrapy.extensions.feedexport] (PID: 101) INFO: Stored bq feed (1000 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:2 2025-11-01 11:44:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:44:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:44:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:44:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4356 pages (at 39 pages/min), scraped 2014 items (at 18 items/min) 2025-11-01 11:45:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:45:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-KIT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:45:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:45:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:45:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4396 pages (at 40 pages/min), scraped 2035 items (at 21 items/min) 2025-11-01 11:46:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:46:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:46:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:46:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-DARKBLUE-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:46:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0015 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0010 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF4060AP-Y already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:46:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:46:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:47:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:47:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4429 pages (at 33 pages/min), scraped 2050 items (at 15 items/min) 2025-11-01 11:47:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GRAPE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:47:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:47:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GRAPE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4441 pages (at 12 pages/min), scraped 2057 items (at 7 items/min) 2025-11-01 11:48:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GRAPE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:48:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:48:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GRAPE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:49:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4441 pages (at 0 pages/min), scraped 2057 items (at 0 items/min) 2025-11-01 11:49:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:17 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:49:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:49:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GRAPE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:42 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:49:42 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 11:49:43 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:49:43 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core.html>: HTTP status code is not handled or not allowed 2025-11-01 11:49:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:49:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:50:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4444 pages (at 3 pages/min), scraped 2057 items (at 0 items/min) 2025-11-01 11:50:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:22 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 11:50:22 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-green.html>: HTTP status code is not handled or not allowed 2025-11-01 11:50:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-PINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:50:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:37 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:50:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:37 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core.html>: HTTP status code is not handled or not allowed 2025-11-01 11:50:38 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 11:50:38 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-hot-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 11:50:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:42 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:50:42 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-red-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 11:50:43 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:50:43 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 11:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-9375-x-0-5-sheet-labels-blockout-217.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:50 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:50:50 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-red.html>: HTTP status code is not handled or not allowed 2025-11-01 11:50:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:50:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:50:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:53 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:50:53 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 11:50:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-5-x-4-5-sheet-labels-blockout-715.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-blockout-720.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:50:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:50:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-125-x-0-5-sheet-labels-blockout-832.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-1-75-sheet-labels-blockout-860.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4461 pages (at 17 pages/min), scraped 2060 items (at 3 items/min) 2025-11-01 11:51:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-3-33-sheet-labels-blockout-863.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-4-5-sheet-labels-white-matte-litho-28.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/10-6-x-1-18-sheet-labels-white-matte-litho-82.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-9375-x-0-5-sheet-labels-blockout-217.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-33-x-3-33-sheet-labels-white-matte-litho-145.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-25-sheet-labels-white-matte-litho-165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-5-x-4-5-sheet-labels-blockout-715.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-blockout-720.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-9-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-125-x-0-5-sheet-labels-blockout-832.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-1-75-sheet-labels-blockout-860.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-3-33-sheet-labels-blockout-863.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-4-5-sheet-labels-white-matte-litho-28.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/10-6-x-1-18-sheet-labels-white-matte-litho-82.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-9375-x-0-5-sheet-labels-blockout-217.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:51:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-33-x-3-33-sheet-labels-white-matte-litho-145.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-25-sheet-labels-white-matte-litho-165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4461 pages (at 0 pages/min), scraped 2060 items (at 0 items/min) 2025-11-01 11:52:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-5-x-4-5-sheet-labels-blockout-715.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-blockout-720.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:52:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-125-x-0-5-sheet-labels-blockout-832.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-1-75-sheet-labels-blockout-860.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-3-33-sheet-labels-blockout-863.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:29 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:52:29 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 11:52:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:52:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-4-5-sheet-labels-white-matte-litho-28.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:52:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:52:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-9375-x-0-5-sheet-labels-blockout-217.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:52:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-25-sheet-labels-white-matte-litho-165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:42 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:52:42 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-dark-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 11:52:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-5-x-4-5-sheet-labels-blockout-715.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-blockout-720.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-3-66-sheet-labels-white-matte-litho-290.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-125-x-0-5-sheet-labels-blockout-832.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-1-75-sheet-labels-blockout-860.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:52:59 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:52:59 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-purple.html>: HTTP status code is not handled or not allowed 2025-11-01 11:53:01 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:53:01 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 11:53:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-3-33-sheet-labels-blockout-863.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4465 pages (at 4 pages/min), scraped 2060 items (at 0 items/min) 2025-11-01 11:53:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-4-5-sheet-labels-white-matte-litho-28.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-blockout-720.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-125-x-0-5-sheet-labels-blockout-832.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-1-75-sheet-labels-blockout-860.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:53:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-3-33-sheet-labels-blockout-863.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:52 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 11:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:52 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-chartreuse.html>: HTTP status code is not handled or not allowed 2025-11-01 11:53:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:53:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:53:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-8-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4487 pages (at 22 pages/min), scraped 2067 items (at 7 items/min) 2025-11-01 11:54:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-1-sheet-labels-white-matte-litho-874.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-9-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-photo-gloss-117.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4521 pages (at 34 pages/min), scraped 2085 items (at 18 items/min) 2025-11-01 11:55:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:55:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:55:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:55:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:55:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4560 pages (at 39 pages/min), scraped 2105 items (at 20 items/min) 2025-11-01 11:56:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:56:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:56:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:56:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:57:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4599 pages (at 39 pages/min), scraped 2124 items (at 19 items/min) 2025-11-01 11:57:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:57:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:57:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:57:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4636 pages (at 37 pages/min), scraped 2141 items (at 17 items/min) 2025-11-01 11:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:58:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:58:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:59:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4675 pages (at 39 pages/min), scraped 2160 items (at 19 items/min) 2025-11-01 11:59:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 11:59:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 11:59:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:00:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4714 pages (at 39 pages/min), scraped 2180 items (at 20 items/min) 2025-11-01 12:00:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:00:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:00:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:00:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:00:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:00:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4754 pages (at 40 pages/min), scraped 2197 items (at 17 items/min) 2025-11-01 12:01:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:01:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:01:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:02:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4792 pages (at 38 pages/min), scraped 2217 items (at 20 items/min) 2025-11-01 12:02:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:02:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:02:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:02:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4832 pages (at 40 pages/min), scraped 2235 items (at 18 items/min) 2025-11-01 12:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:03:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:03:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:03:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4870 pages (at 38 pages/min), scraped 2253 items (at 18 items/min) 2025-11-01 12:04:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:04:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG3050APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTP4060AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:04:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:04:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S921CC already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:04:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4040 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSD0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4904 pages (at 34 pages/min), scraped 2271 items (at 18 items/min) 2025-11-01 12:05:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4020AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4030AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FFD4060 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSC0002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4080A-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4060A-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:05:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT2515AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4936 pages (at 32 pages/min), scraped 2288 items (at 17 items/min) 2025-11-01 12:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:06:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:06:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4080-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:06:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:06:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:06:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:06:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:07:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4961 pages (at 25 pages/min), scraped 2300 items (at 12 items/min) 2025-11-01 12:07:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-2-sheet-labels-blockout-202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-sheet-labels-blockout-210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-7225-x-0-5-sheet-labels-blockout-350.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:07:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:07:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:07:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:07:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-125-x-2-6875-sheet-labels-blockout-870.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:03 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:08:03 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-red.html>: HTTP status code is not handled or not allowed 2025-11-01 12:08:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:08:06 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:08:06 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-green.html>: HTTP status code is not handled or not allowed 2025-11-01 12:08:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:08:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-2-sheet-labels-blockout-202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4967 pages (at 6 pages/min), scraped 2302 items (at 2 items/min) 2025-11-01 12:08:09 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:08:09 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-purple.html>: HTTP status code is not handled or not allowed 2025-11-01 12:08:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-sheet-labels-blockout-210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-7225-x-0-5-sheet-labels-blockout-350.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:17 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:08:17 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-yellow.html>: HTTP status code is not handled or not allowed 2025-11-01 12:08:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:32 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:08:32 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 12:08:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-125-x-2-6875-sheet-labels-blockout-870.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-4-75-sheet-labels-white-matte-litho-26.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-2-sheet-labels-white-matte-litho-87.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-2-sheet-labels-blockout-202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-sheet-labels-blockout-210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-7225-x-0-5-sheet-labels-blockout-350.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:08:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:01 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:09:01 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-green.html>: HTTP status code is not handled or not allowed 2025-11-01 12:09:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4971 pages (at 4 pages/min), scraped 2302 items (at 0 items/min) 2025-11-01 12:09:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-125-x-2-6875-sheet-labels-blockout-870.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-4-75-sheet-labels-white-matte-litho-26.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-2-sheet-labels-white-matte-litho-87.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-2-sheet-labels-blockout-202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-sheet-labels-blockout-210.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-7225-x-0-5-sheet-labels-blockout-350.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:26 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:09:26 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-red.html>: HTTP status code is not handled or not allowed 2025-11-01 12:09:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:09:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-75-sheet-labels-white-matte-litho-275.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-GRAPE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:09:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4050AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:10:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4065AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 4988 pages (at 17 pages/min), scraped 2304 items (at 2 items/min) 2025-11-01 12:10:09 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 12:10:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:09 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-grape-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 12:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4020AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4030AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:10:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:11:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5028 pages (at 40 pages/min), scraped 2326 items (at 22 items/min) 2025-11-01 12:11:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:11:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:11:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:11:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5068 pages (at 40 pages/min), scraped 2347 items (at 21 items/min) 2025-11-01 12:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:12:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:12:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5111 pages (at 43 pages/min), scraped 2365 items (at 18 items/min) 2025-11-01 12:13:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:13:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:13:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:13:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:13:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:13:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5149 pages (at 38 pages/min), scraped 2381 items (at 16 items/min) 2025-11-01 12:14:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:14:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:14:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:14:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:15:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5190 pages (at 41 pages/min), scraped 2402 items (at 21 items/min) 2025-11-01 12:15:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5229 pages (at 39 pages/min), scraped 2422 items (at 20 items/min) 2025-11-01 12:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:16:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5270 pages (at 41 pages/min), scraped 2441 items (at 19 items/min) 2025-11-01 12:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:18:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5308 pages (at 38 pages/min), scraped 2460 items (at 19 items/min) 2025-11-01 12:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:18:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5347 pages (at 39 pages/min), scraped 2477 items (at 17 items/min) 2025-11-01 12:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:17 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5386 pages (at 39 pages/min), scraped 2496 items (at 19 items/min) 2025-11-01 12:20:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:20:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5425 pages (at 39 pages/min), scraped 2516 items (at 20 items/min) 2025-11-01 12:21:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG4090APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG6040APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=9330 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:21:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S906GG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S916CC already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-TOP-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:21:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:22:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5466 pages (at 41 pages/min), scraped 2535 items (at 19 items/min) 2025-11-01 12:22:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSB0003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:22:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF4060AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4100AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0007 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4030A-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3010AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:22:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:23:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5503 pages (at 37 pages/min), scraped 2551 items (at 16 items/min) 2025-11-01 12:23:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:23:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:23:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:23:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:23:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:23:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-5-x-4-5-sheet-labels-blockout-20.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5514 pages (at 11 pages/min), scraped 2558 items (at 7 items/min) 2025-11-01 12:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:24:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:24:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-5-x-4-5-sheet-labels-blockout-20.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:24:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-blockout-77.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:24:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:24:59 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:24:59 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-pink-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 12:25:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:25:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5515 pages (at 1 pages/min), scraped 2558 items (at 0 items/min) 2025-11-01 12:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-5-x-4-5-sheet-labels-blockout-20.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-4375-x-0-667-sheet-labels-blockout-125.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:25:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:28 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:25:28 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-yellow-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 12:25:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:25:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-blockout-77.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-52-x-4-52-sheet-labels-blockout-155.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-25-sheet-labels-blockout-165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:25:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-5-x-4-5-sheet-labels-blockout-20.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-4375-x-0-667-sheet-labels-blockout-125.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-4-75-sheet-labels-blockout-203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5516 pages (at 1 pages/min), scraped 2558 items (at 0 items/min) 2025-11-01 12:26:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-sheet-labels-blockout-214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-blockout-77.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-52-x-4-52-sheet-labels-blockout-155.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-25-sheet-labels-blockout-165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:24 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:26:24 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLORANGE>: HTTP status code is not handled or not allowed 2025-11-01 12:26:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:26:27 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:26:27 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-polypropylene.html>: HTTP status code is not handled or not allowed 2025-11-01 12:26:29 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:26:29 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-HOTPINK>: HTTP status code is not handled or not allowed 2025-11-01 12:26:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-5-x-4-5-sheet-labels-blockout-20.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-4375-x-0-667-sheet-labels-blockout-125.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:34 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:26:35 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-chartreuse.html>: HTTP status code is not handled or not allowed 2025-11-01 12:26:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-4-75-sheet-labels-blockout-203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:26:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:26:41 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:26:41 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-grape.html>: HTTP status code is not handled or not allowed 2025-11-01 12:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:43 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:26:44 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-green.html>: HTTP status code is not handled or not allowed 2025-11-01 12:26:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-sheet-labels-blockout-214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-blockout-77.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-52-x-4-52-sheet-labels-blockout-155.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-25-sheet-labels-blockout-165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:51 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:26:51 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-purple.html>: HTTP status code is not handled or not allowed 2025-11-01 12:26:53 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:26:53 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 12:26:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:26:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:27:05 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:27:05 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 12:27:07 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:27:07 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-5-x-4-5-sheet-labels-blockout-20.html>: HTTP status code is not handled or not allowed 2025-11-01 12:27:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5526 pages (at 10 pages/min), scraped 2558 items (at 0 items/min) 2025-11-01 12:27:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-4375-x-0-667-sheet-labels-blockout-125.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-4-75-sheet-labels-blockout-203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:14 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:27:14 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-fluorescent-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 12:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-white-matte-litho-106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-5-x-0-75-sheet-labels-white-matte-litho-181.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-sheet-labels-blockout-214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-blockout-77.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-52-x-4-52-sheet-labels-blockout-155.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-25-sheet-labels-blockout-165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-9375-x-3-9375-sheet-labels-white-matte-litho-410.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:32 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:27:32 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-RED>: HTTP status code is not handled or not allowed 2025-11-01 12:27:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-65-x-4-65-sheet-labels-white-matte-litho-490.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-sheet-labels-white-matte-litho-500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-4375-x-0-667-sheet-labels-blockout-125.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-2-sheet-labels-white-matte-litho-730.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-4-75-sheet-labels-blockout-203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:27:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-white-matte-litho-106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-5-x-0-75-sheet-labels-white-matte-litho-181.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-sheet-labels-blockout-214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:01 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:28:01 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-2-sheet-labels-blockout-77.html>: HTTP status code is not handled or not allowed 2025-11-01 12:28:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:28:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-52-x-4-52-sheet-labels-blockout-155.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-25-sheet-labels-blockout-165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-9375-x-3-9375-sheet-labels-white-matte-litho-410.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5529 pages (at 3 pages/min), scraped 2558 items (at 0 items/min) 2025-11-01 12:28:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-0-946-sheet-labels-white-matte-litho-840.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:13 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:28:13 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-hot-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 12:28:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:28:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:28:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-65-x-4-65-sheet-labels-white-matte-litho-490.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-sheet-labels-white-matte-litho-500.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:23 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:28:24 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-4375-x-0-667-sheet-labels-blockout-125.html>: HTTP status code is not handled or not allowed 2025-11-01 12:28:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-2-sheet-labels-white-matte-litho-730.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-4-75-sheet-labels-blockout-203.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-9-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-white-matte-litho-106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-5-x-0-75-sheet-labels-white-matte-litho-181.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-sheet-labels-blockout-214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-9-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:28:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-9-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:28:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5625-x-1-8125-sheet-labels-photo-gloss-625.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-65-x-4-65-sheet-labels-white-matte-litho-490.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-2-sheet-labels-white-matte-litho-730.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:29:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-9-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5542 pages (at 13 pages/min), scraped 2558 items (at 0 items/min) 2025-11-01 12:29:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-5-x-0-75-sheet-labels-white-matte-litho-181.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:13 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:13 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-sheet-labels-blockout-214.html>: HTTP status code is not handled or not allowed 2025-11-01 12:29:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-CHARTREUSE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-5-sheet-labels-white-mid-gloss-104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-9-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:29:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-65-x-4-65-sheet-labels-white-matte-litho-490.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-9-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-2-sheet-labels-white-matte-litho-730.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:29:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-CHARTREUSE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-5-sheet-labels-white-mid-gloss-104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-white-mid-gloss-695.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:30:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5556 pages (at 14 pages/min), scraped 2561 items (at 3 items/min) 2025-11-01 12:30:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-white-mid-gloss-720.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-2-sheet-labels-white-premium-high-gloss-175.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-65-x-4-65-sheet-labels-white-matte-litho-490.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-x-1-sheet-labels-white-premium-high-gloss-187.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:30:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:29 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:30:29 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-green.html>: HTTP status code is not handled or not allowed 2025-11-01 12:30:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-sheet-labels-white-premium-high-gloss-198.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-875-x-4-312-sheet-labels-white-premium-high-gloss-680.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:52 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:30:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:52 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-65-x-4-65-sheet-labels-white-matte-litho-490.html>: HTTP status code is not handled or not allowed 2025-11-01 12:30:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:30:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-5-x-5-5-sheet-labels-clear-matte-41.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-1-75-sheet-labels-white-premium-high-gloss-860.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-sheet-labels-white-premium-high-gloss-198.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5572 pages (at 16 pages/min), scraped 2566 items (at 5 items/min) 2025-11-01 12:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-25-sheet-labels-clear-matte-192.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:31:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-1-5-sheet-labels-clear-matte-400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-875-x-4-312-sheet-labels-white-premium-high-gloss-680.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-75-x-2-5-sheet-labels-clear-matte-867.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:31:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-5-375-sheet-labels-clear-matte-872.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:31:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-1-75-sheet-labels-white-premium-high-gloss-860.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:31:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-1-5-sheet-labels-clear-matte-400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-125-x-2-6875-sheet-labels-crystal-clear-870.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-875-x-4-312-sheet-labels-white-premium-high-gloss-680.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5588 pages (at 16 pages/min), scraped 2575 items (at 9 items/min) 2025-11-01 12:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-sheet-labels-red-215.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-red-142.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-5-375-sheet-labels-clear-matte-872.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-5-25-sheet-labels-red-816.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-0-666-sheet-labels-red-891.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-0-5-sheet-labels-gray-55.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-25-sheet-labels-red-878.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:30 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:32:30 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165OP>: HTTP status code is not handled or not allowed 2025-11-01 12:32:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:32:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-1-5-sheet-labels-clear-matte-400.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:36 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:32:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:37 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410ML>: HTTP status code is not handled or not allowed 2025-11-01 12:32:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-875-x-4-312-sheet-labels-white-premium-high-gloss-680.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-sheet-labels-red-215.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:32:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-5-25-sheet-labels-red-816.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-0-666-sheet-labels-red-891.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-25-sheet-labels-red-878.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-sheet-labels-gray-120.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5607 pages (at 19 pages/min), scraped 2585 items (at 10 items/min) 2025-11-01 12:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:33:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:33:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5648 pages (at 41 pages/min), scraped 2604 items (at 19 items/min) 2025-11-01 12:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:34:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:34:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:34:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:34:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5686 pages (at 38 pages/min), scraped 2619 items (at 15 items/min) 2025-11-01 12:35:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:35:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:35:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:35:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:35:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5726 pages (at 40 pages/min), scraped 2637 items (at 18 items/min) 2025-11-01 12:36:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:36:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:37:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:37:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5765 pages (at 39 pages/min), scraped 2657 items (at 20 items/min) 2025-11-01 12:37:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:37:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5802 pages (at 37 pages/min), scraped 2677 items (at 20 items/min) 2025-11-01 12:38:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:38:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:38:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5843 pages (at 41 pages/min), scraped 2696 items (at 19 items/min) 2025-11-01 12:39:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5882 pages (at 39 pages/min), scraped 2714 items (at 18 items/min) 2025-11-01 12:40:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:40:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:40:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:40:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTF4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5920 pages (at 38 pages/min), scraped 2733 items (at 19 items/min) 2025-11-01 12:41:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S913BB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3020AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF3050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4040AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4013AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5957 pages (at 37 pages/min), scraped 2753 items (at 20 items/min) 2025-11-01 12:42:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060A-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT2015AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF3050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4013AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:42:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT2015AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5973 pages (at 16 pages/min), scraped 2760 items (at 7 items/min) 2025-11-01 12:43:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF3050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4013AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-0-75-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT2015AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF3050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:43:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:43:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-0-75-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5973 pages (at 0 pages/min), scraped 2760 items (at 0 items/min) 2025-11-01 12:44:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT2015AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF3050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-0-75-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:45 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:44:45 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-dark-blue-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 12:44:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:44:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT2015AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:51 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:44:51 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-orange-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 12:44:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:44:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:44:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-6-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:44:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF3050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:45:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:45:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5975 pages (at 2 pages/min), scraped 2760 items (at 0 items/min) 2025-11-01 12:45:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:30 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:45:30 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4030AP>: HTTP status code is not handled or not allowed 2025-11-01 12:45:31 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 12:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:31 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 12:45:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:35 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:45:35 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-red.html>: HTTP status code is not handled or not allowed 2025-11-01 12:45:37 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:45:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:37 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-yellow.html>: HTTP status code is not handled or not allowed 2025-11-01 12:45:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:42 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:45:42 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-purple.html>: HTTP status code is not handled or not allowed 2025-11-01 12:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:46 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:45:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:46 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0001>: HTTP status code is not handled or not allowed 2025-11-01 12:45:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:45:58 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 12:45:58 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-hot-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 12:45:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5994 pages (at 19 pages/min), scraped 2767 items (at 7 items/min) 2025-11-01 12:46:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:21 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:46:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:21 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 12:46:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:46:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-5-x-4-125-sheet-labels-blockout-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:46:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:46:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:46:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:47:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:47:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-5-x-4-125-sheet-labels-blockout-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5998 pages (at 4 pages/min), scraped 2769 items (at 2 items/min) 2025-11-01 12:47:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:47:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:17 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:47:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-2-5-sheet-labels-blockout-140.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:47:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:47:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-375-sheet-labels-blockout-285.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-5-x-4-125-sheet-labels-blockout-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-0-5-sheet-labels-blockout-605.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-2-5-sheet-labels-blockout-140.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-7656-x-4-7656-sheet-labels-blockout-835.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:47:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 5998 pages (at 0 pages/min), scraped 2769 items (at 0 items/min) 2025-11-01 12:48:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-375-x-1-25-sheet-labels-blockout-884.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-375-sheet-labels-blockout-285.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-5-x-4-125-sheet-labels-blockout-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:21 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:48:21 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-yellow.html>: HTTP status code is not handled or not allowed 2025-11-01 12:48:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-0-5-sheet-labels-blockout-605.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:25 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:48:25 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 12:48:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-2-5-sheet-labels-blockout-140.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-7656-x-4-7656-sheet-labels-blockout-835.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-59375-x-1-3-sheet-labels-white-matte-litho-76.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:38 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:48:38 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-polypropylene.html>: HTTP status code is not handled or not allowed 2025-11-01 12:48:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-375-x-1-25-sheet-labels-blockout-884.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-375-sheet-labels-blockout-285.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-5-x-4-125-sheet-labels-blockout-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:50 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:48:50 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blockout.html>: HTTP status code is not handled or not allowed 2025-11-01 12:48:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:53 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:48:53 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-GREEN-XL>: HTTP status code is not handled or not allowed 2025-11-01 12:48:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-1-25-sheet-labels-white-matte-litho-240.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-0-5-sheet-labels-blockout-605.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-5-sheet-labels-white-matte-litho-255.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:00 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:49:00 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-fluorescent-green.html>: HTTP status code is not handled or not allowed 2025-11-01 12:49:01 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:49:01 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-5-direct-thermal-labels-3-core-top-coated.html>: HTTP status code is not handled or not allowed 2025-11-01 12:49:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-2-5-sheet-labels-blockout-140.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-7656-x-4-7656-sheet-labels-blockout-835.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-59375-x-1-3-sheet-labels-white-matte-litho-76.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6005 pages (at 7 pages/min), scraped 2769 items (at 0 items/min) 2025-11-01 12:49:08 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:49:08 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-green.html>: HTTP status code is not handled or not allowed 2025-11-01 12:49:10 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:49:10 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 12:49:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-9-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-375-x-1-25-sheet-labels-blockout-884.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-375-sheet-labels-blockout-285.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:23 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:49:23 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/5-5-x-4-125-sheet-labels-blockout-9.html>: HTTP status code is not handled or not allowed 2025-11-01 12:49:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-4375-x-5-4531-sheet-labels-white-matte-litho-306.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:26 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:49:26 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-dark-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 12:49:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-1-5-sheet-labels-white-matte-litho-390.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-1-25-sheet-labels-white-matte-litho-240.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:49:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-0-5-sheet-labels-blockout-605.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-5-sheet-labels-white-matte-litho-255.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-64-x-0-42-sheet-labels-white-matte-litho-620.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-2-5-sheet-labels-blockout-140.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-7656-x-4-7656-sheet-labels-blockout-835.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-59375-x-1-3-sheet-labels-white-matte-litho-76.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5625-sheet-labels-white-matte-litho-785.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:46 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:49:47 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-yellow.html>: HTTP status code is not handled or not allowed 2025-11-01 12:49:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-9-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:51 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:49:51 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-orange-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 12:49:53 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:49:53 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-FLORANGE>: HTTP status code is not handled or not allowed 2025-11-01 12:49:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-375-x-1-25-sheet-labels-blockout-884.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-375-sheet-labels-blockout-285.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:49:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-8-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-4375-x-5-4531-sheet-labels-white-matte-litho-306.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:50:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-1-5-sheet-labels-white-matte-litho-390.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-1-25-sheet-labels-white-matte-litho-240.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-9-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6012 pages (at 7 pages/min), scraped 2769 items (at 0 items/min) 2025-11-01 12:50:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-0-5-sheet-labels-blockout-605.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-5-sheet-labels-white-matte-litho-255.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-64-x-0-42-sheet-labels-white-matte-litho-620.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:50:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:50:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:16 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:50:16 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-5-x-2-5-sheet-labels-blockout-140.html>: HTTP status code is not handled or not allowed 2025-11-01 12:50:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-7656-x-4-7656-sheet-labels-blockout-835.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-59375-x-1-3-sheet-labels-white-matte-litho-76.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5625-sheet-labels-white-matte-litho-785.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-9-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-9-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-x-1-sheet-labels-photo-gloss-187.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-3-5-sheet-labels-photo-gloss-200.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-375-x-1-25-sheet-labels-blockout-884.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:33 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:50:33 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-5-x-1-375-sheet-labels-blockout-285.html>: HTTP status code is not handled or not allowed 2025-11-01 12:50:34 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 12:50:35 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-gray-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 12:50:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-8-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:50:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:50:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4090AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:51:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:51:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:51:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6033 pages (at 21 pages/min), scraped 2769 items (at 0 items/min) 2025-11-01 12:51:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:51:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:51:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6074 pages (at 41 pages/min), scraped 2793 items (at 24 items/min) 2025-11-01 12:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:52:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:52:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:52:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:53:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6114 pages (at 40 pages/min), scraped 2814 items (at 21 items/min) 2025-11-01 12:53:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:53:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:54:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6152 pages (at 38 pages/min), scraped 2830 items (at 16 items/min) 2025-11-01 12:54:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:54:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:54:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:54:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:54:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:54:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:54:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6190 pages (at 38 pages/min), scraped 2845 items (at 15 items/min) 2025-11-01 12:55:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:55:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:55:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:55:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6231 pages (at 41 pages/min), scraped 2867 items (at 22 items/min) 2025-11-01 12:56:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:56:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:56:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:56:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6271 pages (at 40 pages/min), scraped 2890 items (at 23 items/min) 2025-11-01 12:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:57:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:57:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6311 pages (at 40 pages/min), scraped 2908 items (at 18 items/min) 2025-11-01 12:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:58:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:58:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:58:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6349 pages (at 38 pages/min), scraped 2924 items (at 16 items/min) 2025-11-01 12:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:59:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 12:59:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 12:59:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6390 pages (at 41 pages/min), scraped 2944 items (at 20 items/min) 2025-11-01 13:00:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6428 pages (at 38 pages/min), scraped 2965 items (at 21 items/min) 2025-11-01 13:01:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:01:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:01:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:02:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6469 pages (at 41 pages/min), scraped 2984 items (at 19 items/min) 2025-11-01 13:02:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG30100APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG40120APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:02:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=Z*0433*985*W*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:02:59 [py.warnings] (PID: 101) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 13:03:00 [scrapy.extensions.feedexport] (PID: 101) INFO: Stored bq feed (1000 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:3 2025-11-01 13:03:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6510 pages (at 41 pages/min), scraped 3003 items (at 19 items/min) 2025-11-01 13:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTP4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTPF4010AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTAG4065AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:03:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:03:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S918DD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSD0003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:03:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:04:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:04:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:04:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6550 pages (at 40 pages/min), scraped 3021 items (at 18 items/min) 2025-11-01 13:04:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSC0004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:04:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:04:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT6040AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:04:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:04:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:04:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:04:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:04:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:04:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:04:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:04:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:04:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:04:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:04:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:04:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT2060AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4030-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6581 pages (at 31 pages/min), scraped 3038 items (at 17 items/min) 2025-11-01 13:05:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:05:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT2060AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:05:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT2060AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-3-core-nonperforated-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6599 pages (at 18 pages/min), scraped 3045 items (at 7 items/min) 2025-11-01 13:06:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:06:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT2060AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:06:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:06:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6599 pages (at 0 pages/min), scraped 3045 items (at 0 items/min) 2025-11-01 13:07:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT2060AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:29 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 13:07:30 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-fluorescent-green.html>: HTTP status code is not handled or not allowed 2025-11-01 13:07:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-blockout-110.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:56 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 13:07:56 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-polypropylene.html>: HTTP status code is not handled or not allowed 2025-11-01 13:07:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT2060AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:07:59 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 13:07:59 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-hot-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 13:08:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6602 pages (at 3 pages/min), scraped 3045 items (at 0 items/min) 2025-11-01 13:08:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:08:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:08:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3580AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:08:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6633 pages (at 31 pages/min), scraped 3059 items (at 14 items/min) 2025-11-01 13:09:09 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 13:09:09 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-dark-blue-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 13:09:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:09:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:09:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:09:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:09:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6670 pages (at 37 pages/min), scraped 3078 items (at 19 items/min) 2025-11-01 13:10:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:15 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 13:10:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:15 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-hot-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 13:10:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:10:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:10:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:10:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:11:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6713 pages (at 43 pages/min), scraped 3099 items (at 21 items/min) 2025-11-01 13:11:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:11:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:11:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:11:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:11:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:12:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:12:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:12:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6750 pages (at 37 pages/min), scraped 3117 items (at 18 items/min) 2025-11-01 13:12:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:12:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:12:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6789 pages (at 39 pages/min), scraped 3133 items (at 16 items/min) 2025-11-01 13:13:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:13:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:13:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:13:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:13:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:13:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:13:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6829 pages (at 40 pages/min), scraped 3154 items (at 21 items/min) 2025-11-01 13:14:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:14:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:14:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6869 pages (at 40 pages/min), scraped 3173 items (at 19 items/min) 2025-11-01 13:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6907 pages (at 38 pages/min), scraped 3191 items (at 18 items/min) 2025-11-01 13:16:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:16:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6948 pages (at 41 pages/min), scraped 3211 items (at 20 items/min) 2025-11-01 13:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 6986 pages (at 38 pages/min), scraped 3226 items (at 15 items/min) 2025-11-01 13:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:18:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7027 pages (at 41 pages/min), scraped 3248 items (at 22 items/min) 2025-11-01 13:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:19:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7065 pages (at 38 pages/min), scraped 3267 items (at 19 items/min) 2025-11-01 13:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:20:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG4040APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG4015APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:21:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7106 pages (at 41 pages/min), scraped 3285 items (at 18 items/min) 2025-11-01 13:21:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:21:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:21:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:21:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:21:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:21:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:21:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060AP-ORG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4080AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSC0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:21:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7141 pages (at 35 pages/min), scraped 3302 items (at 17 items/min) 2025-11-01 13:22:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:22:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:22:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:22:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:22:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:22:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:23:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7166 pages (at 25 pages/min), scraped 3318 items (at 16 items/min) 2025-11-01 13:23:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:23:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:23:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-4375-sheet-labels-blockout-86.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:23:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:23:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-11-sheet-labels-blockout-93.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7166 pages (at 0 pages/min), scraped 3318 items (at 0 items/min) 2025-11-01 13:24:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-4375-sheet-labels-blockout-86.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:24:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:24:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-sheet-labels-blockout-191.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:24:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:39 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 13:24:39 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-brown-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 13:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-4375-sheet-labels-blockout-86.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-25-sheet-labels-blockout-192.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:24:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-sheet-labels-blockout-191.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:24:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-sheet-labels-blockout-280.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:25:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7167 pages (at 1 pages/min), scraped 3318 items (at 0 items/min) 2025-11-01 13:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-75-x-2-sheet-labels-blockout-730.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-625-x-0-375-sheet-labels-blockout-838.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-4375-sheet-labels-blockout-86.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:25:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-RED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:25:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4010AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:26:03 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 13:26:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:26:03 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-blockout.html>: HTTP status code is not handled or not allowed 2025-11-01 13:26:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:26:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7193 pages (at 26 pages/min), scraped 3323 items (at 5 items/min) 2025-11-01 13:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:26:19 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 13:26:20 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-blockout.html>: HTTP status code is not handled or not allowed 2025-11-01 13:26:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4065AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:26:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:26:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:26:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:26:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:26:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:26:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7229 pages (at 36 pages/min), scraped 3342 items (at 19 items/min) 2025-11-01 13:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:27:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:27:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:27:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:27:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:28:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7270 pages (at 41 pages/min), scraped 3364 items (at 22 items/min) 2025-11-01 13:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:28:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:28:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:28:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:29:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:29:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7310 pages (at 40 pages/min), scraped 3384 items (at 20 items/min) 2025-11-01 13:29:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:29:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:29:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7351 pages (at 41 pages/min), scraped 3399 items (at 15 items/min) 2025-11-01 13:30:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:30:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:30:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7389 pages (at 38 pages/min), scraped 3416 items (at 17 items/min) 2025-11-01 13:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:31:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:31:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:31:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:31:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:32:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7429 pages (at 40 pages/min), scraped 3438 items (at 22 items/min) 2025-11-01 13:32:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:32:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:32:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7468 pages (at 39 pages/min), scraped 3458 items (at 20 items/min) 2025-11-01 13:33:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:33:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7507 pages (at 39 pages/min), scraped 3476 items (at 18 items/min) 2025-11-01 13:34:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:34:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:34:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:34:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:34:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:34:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7546 pages (at 39 pages/min), scraped 3492 items (at 16 items/min) 2025-11-01 13:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:35:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:35:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:35:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7588 pages (at 42 pages/min), scraped 3513 items (at 21 items/min) 2025-11-01 13:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=D*0400*1182*W*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:36:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=Z*0327*1476*X*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4010AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-CHA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=800*160*200*110*H already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=9402 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=9404 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7626 pages (at 38 pages/min), scraped 3535 items (at 22 items/min) 2025-11-01 13:37:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:37:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3050AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:37:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:37:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT6040A-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4080AP-UPS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7659 pages (at 33 pages/min), scraped 3551 items (at 16 items/min) 2025-11-01 13:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-PINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:38:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:38:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7687 pages (at 28 pages/min), scraped 3566 items (at 15 items/min) 2025-11-01 13:39:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4065AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-5-sheet-labels-blockout-211.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:39:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-fluorescent-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4065AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7692 pages (at 5 pages/min), scraped 3567 items (at 1 items/min) 2025-11-01 13:40:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-5-sheet-labels-blockout-211.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-fluorescent-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:21 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 13:40:21 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-hot-pink-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 13:40:22 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 13:40:22 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-yellow.html>: HTTP status code is not handled or not allowed 2025-11-01 13:40:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4065AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-5-sheet-labels-blockout-211.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:53 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 13:40:53 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-green.html>: HTTP status code is not handled or not allowed 2025-11-01 13:40:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-fluorescent-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-5-sheet-labels-blockout-325.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:40:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4065AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7695 pages (at 3 pages/min), scraped 3567 items (at 0 items/min) 2025-11-01 13:41:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:41:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:12 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 13:41:12 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-red.html>: HTTP status code is not handled or not allowed 2025-11-01 13:41:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-5-x-2-0875-sheet-labels-blockout-435.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-5-sheet-labels-blockout-211.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5625-x-1-8125-sheet-labels-blockout-625.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-fluorescent-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-5-sheet-labels-blockout-325.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:41:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4065AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-5-25-sheet-labels-blockout-816.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-25-x-0-5-sheet-labels-blockout-829.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-5-x-2-0875-sheet-labels-blockout-435.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:56 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 13:41:56 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-orange-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 13:41:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:41:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7701 pages (at 6 pages/min), scraped 3568 items (at 1 items/min) 2025-11-01 13:42:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:42:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040A-FLGREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:19 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 13:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:20 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-chartreuse.html>: HTTP status code is not handled or not allowed 2025-11-01 13:42:25 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 13:42:25 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 13:42:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:42:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:42:31 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 13:42:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:31 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 13:42:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:42:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:42:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:43:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7742 pages (at 41 pages/min), scraped 3583 items (at 15 items/min) 2025-11-01 13:43:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:43:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7783 pages (at 41 pages/min), scraped 3605 items (at 22 items/min) 2025-11-01 13:44:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:44:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:44:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7821 pages (at 38 pages/min), scraped 3622 items (at 17 items/min) 2025-11-01 13:45:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:45:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:45:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:45:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:45:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:45:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7860 pages (at 39 pages/min), scraped 3638 items (at 16 items/min) 2025-11-01 13:46:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:46:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:46:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:46:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:46:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:46:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7900 pages (at 40 pages/min), scraped 3661 items (at 23 items/min) 2025-11-01 13:47:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:47:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:47:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:47:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7941 pages (at 41 pages/min), scraped 3681 items (at 20 items/min) 2025-11-01 13:48:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 7980 pages (at 39 pages/min), scraped 3699 items (at 18 items/min) 2025-11-01 13:49:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8020 pages (at 40 pages/min), scraped 3718 items (at 19 items/min) 2025-11-01 13:50:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-2-2-sheet-labels-fluorescent-pink-345.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-25-x-0-5-sheet-labels-fluorescent-pink-745.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-0-5-sheet-labels-fluorescent-pink-883.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-4375-x-0-667-sheet-labels-fluorescent-green-126.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-5-x-2-sheet-labels-fluorescent-green-207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:50:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-75-sheet-labels-fluorescent-green-275.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-6-x-1-77-sheet-labels-fluorescent-green-430.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8051 pages (at 31 pages/min), scraped 3735 items (at 17 items/min) 2025-11-01 13:51:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-75-sheet-labels-fluorescent-green-275.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-sheet-labels-fluorescent-yellow-198.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-sheet-labels-fluorescent-yellow-214.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-875-x-2-6875-sheet-labels-fluorescent-yellow-650.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:51:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:51:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:51:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-5-sheet-labels-fluorescent-orange-104.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:52:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8079 pages (at 28 pages/min), scraped 3747 items (at 12 items/min) 2025-11-01 13:52:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:52:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:53:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8119 pages (at 40 pages/min), scraped 3764 items (at 17 items/min) 2025-11-01 13:53:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG4010APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:53:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=Z*0238*1476*W*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FFT4020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=800*190*300*1000*H already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:53:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:53:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S919DD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-POLY-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:54:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8156 pages (at 37 pages/min), scraped 3782 items (at 18 items/min) 2025-11-01 13:54:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4080AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4060A-TOP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT2010AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:54:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:54:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:54:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:54:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:54:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT3020AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8185 pages (at 29 pages/min), scraped 3799 items (at 17 items/min) 2025-11-01 13:55:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:55:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:55:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:56:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:56:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8216 pages (at 31 pages/min), scraped 3814 items (at 15 items/min) 2025-11-01 13:56:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4070-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:17 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:56:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:56:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:56:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:56:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:57:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:57:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8242 pages (at 26 pages/min), scraped 3825 items (at 11 items/min) 2025-11-01 13:57:10 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 13:57:10 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-grape.html>: HTTP status code is not handled or not allowed 2025-11-01 13:57:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040A-GRAPE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040A-CHARTREUSE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:27 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 13:57:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:27 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 13:57:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:57:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:57:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:57:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8278 pages (at 36 pages/min), scraped 3841 items (at 16 items/min) 2025-11-01 13:58:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:58:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:29 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 13:58:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:29 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-fluorescent-red.html>: HTTP status code is not handled or not allowed 2025-11-01 13:58:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:58:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:58:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 13:59:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8312 pages (at 34 pages/min), scraped 3857 items (at 16 items/min) 2025-11-01 13:59:09 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 13:59:09 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-hot-pink-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 13:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:27 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 13:59:27 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-brown.html>: HTTP status code is not handled or not allowed 2025-11-01 13:59:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 13:59:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:00:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8353 pages (at 41 pages/min), scraped 3876 items (at 19 items/min) 2025-11-01 14:00:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:00:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:00:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8392 pages (at 39 pages/min), scraped 3895 items (at 19 items/min) 2025-11-01 14:01:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:01:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8433 pages (at 41 pages/min), scraped 3917 items (at 22 items/min) 2025-11-01 14:02:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:02:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:03:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8472 pages (at 39 pages/min), scraped 3934 items (at 17 items/min) 2025-11-01 14:03:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:03:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:03:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:03:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8510 pages (at 38 pages/min), scraped 3952 items (at 18 items/min) 2025-11-01 14:04:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:04:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:04:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8550 pages (at 40 pages/min), scraped 3973 items (at 21 items/min) 2025-11-01 14:05:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:05:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:05:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:05:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:05:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-4375-sheet-labels-fluorescent-red-85.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8587 pages (at 37 pages/min), scraped 3992 items (at 19 items/min) 2025-11-01 14:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-25-sheet-labels-fluorescent-red-165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-1-sheet-labels-fluorescent-pink-16.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-x-1-sheet-labels-fluorescent-pink-630.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-4-875-sheet-labels-fluorescent-pink-861.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:06:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:06:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:06:49 [py.warnings] (PID: 101) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 14:06:50 [scrapy.extensions.feedexport] (PID: 101) INFO: Stored bq feed (1000 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:4 2025-11-01 14:06:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8619 pages (at 32 pages/min), scraped 4007 items (at 15 items/min) 2025-11-01 14:07:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:07:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:07:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8658 pages (at 39 pages/min), scraped 4026 items (at 19 items/min) 2025-11-01 14:08:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:08:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:08:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:08:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:08:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:09:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8696 pages (at 38 pages/min), scraped 4042 items (at 16 items/min) 2025-11-01 14:09:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:09:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:09:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=800*160*300*110*H already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=800*190*300*110*H already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-2-375-sheet-labels-685.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-0-75-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-direct-thermal-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:09:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:09:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4025AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT5030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4065AP-2X already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-inch-x-1-75-inch-blank-hobart-labels.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8730 pages (at 34 pages/min), scraped 4058 items (at 16 items/min) 2025-11-01 14:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:10:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT2010AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:10:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3020A-TOP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-poly.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-CHARTREUSE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:10:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-CHARTREUSE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8768 pages (at 38 pages/min), scraped 4077 items (at 19 items/min) 2025-11-01 14:11:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-CHARTREUSE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3060AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-poly.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-PINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4010AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:11:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4010AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-poly.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:12:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8798 pages (at 30 pages/min), scraped 4091 items (at 14 items/min) 2025-11-01 14:12:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4015AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:29 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 14:12:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:29 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core.html>: HTTP status code is not handled or not allowed 2025-11-01 14:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-poly.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:12:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-white-matte-litho-71.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:12:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:12:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4065AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8832 pages (at 34 pages/min), scraped 4109 items (at 18 items/min) 2025-11-01 14:13:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-poly.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:46 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 14:13:46 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-yellow.html>: HTTP status code is not handled or not allowed 2025-11-01 14:13:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:13:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:13:55 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 14:13:55 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-poly.html>: HTTP status code is not handled or not allowed 2025-11-01 14:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:14:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8868 pages (at 36 pages/min), scraped 4125 items (at 16 items/min) 2025-11-01 14:14:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:14:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:14:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:14:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:50 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 14:14:50 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-yellow-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 14:14:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:14:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8909 pages (at 41 pages/min), scraped 4141 items (at 16 items/min) 2025-11-01 14:15:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8946 pages (at 37 pages/min), scraped 4159 items (at 18 items/min) 2025-11-01 14:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:16:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:16:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 8983 pages (at 37 pages/min), scraped 4179 items (at 20 items/min) 2025-11-01 14:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:17:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:17:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:17:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9025 pages (at 42 pages/min), scraped 4198 items (at 19 items/min) 2025-11-01 14:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9065 pages (at 40 pages/min), scraped 4218 items (at 20 items/min) 2025-11-01 14:19:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:19:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:19:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:19:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9104 pages (at 39 pages/min), scraped 4234 items (at 16 items/min) 2025-11-01 14:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-5-375-sheet-labels-fluorescent-red-872.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:20:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-5-x-2-0875-sheet-labels-fluorescent-pink-435.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-3125-sheet-labels-fluorescent-pink-655.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:20:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-7-sheet-labels-fluorescent-pink-817.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-sheet-labels-fluorescent-pink-885.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-25-sheet-labels-fluorescent-green-192.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9133 pages (at 29 pages/min), scraped 4250 items (at 16 items/min) 2025-11-01 14:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-3-66-sheet-labels-fluorescent-green-290.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:21:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:21:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:21:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:21:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:22:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:22:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9167 pages (at 34 pages/min), scraped 4265 items (at 15 items/min) 2025-11-01 14:22:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:22:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:22:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:22:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:22:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:23:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9205 pages (at 38 pages/min), scraped 4283 items (at 18 items/min) 2025-11-01 14:23:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:23:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG4060APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG2015APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG4000APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG6090APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9245 pages (at 40 pages/min), scraped 4301 items (at 18 items/min) 2025-11-01 14:24:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:24:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:24:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:24:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:24:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSD0002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4090AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4060AP*Y already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9281 pages (at 36 pages/min), scraped 4320 items (at 19 items/min) 2025-11-01 14:25:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT225125AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-3-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-3-core-nonperforated-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4090AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT225125AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:26:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9292 pages (at 11 pages/min), scraped 4325 items (at 5 items/min) 2025-11-01 14:26:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:26:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-3-core-nonperforated-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4090AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:26:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:26:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT225125AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-0-75-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-3-core-nonperforated-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4090AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:27:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9292 pages (at 0 pages/min), scraped 4325 items (at 0 items/min) 2025-11-01 14:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT225125AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-0-75-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-3-core-nonperforated-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4090AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0004 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT225125AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:27:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-PINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9294 pages (at 2 pages/min), scraped 4325 items (at 0 items/min) 2025-11-01 14:28:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060A-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:28:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:25 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:28:25 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-red.html>: HTTP status code is not handled or not allowed 2025-11-01 14:28:27 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:28:27 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129W>: HTTP status code is not handled or not allowed 2025-11-01 14:28:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT225125AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:33 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:28:34 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213W>: HTTP status code is not handled or not allowed 2025-11-01 14:28:35 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:28:35 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275W>: HTTP status code is not handled or not allowed 2025-11-01 14:28:37 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:28:37 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-fluorescent-green.html>: HTTP status code is not handled or not allowed 2025-11-01 14:28:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:40 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:28:40 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 14:28:41 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:28:41 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745W>: HTTP status code is not handled or not allowed 2025-11-01 14:28:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:28:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-PINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060A-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:53 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:28:53 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0003>: HTTP status code is not handled or not allowed 2025-11-01 14:28:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:55 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:28:55 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 14:28:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:28:58 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:28:59 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-GRAPE>: HTTP status code is not handled or not allowed 2025-11-01 14:29:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-fanfold-labels-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:05 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:29:05 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT225125AP-TOP>: HTTP status code is not handled or not allowed 2025-11-01 14:29:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9309 pages (at 15 pages/min), scraped 4328 items (at 3 items/min) 2025-11-01 14:29:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:11 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:29:11 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-BROWN>: HTTP status code is not handled or not allowed 2025-11-01 14:29:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:29:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:16 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:29:16 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-polypropylene.html>: HTTP status code is not handled or not allowed 2025-11-01 14:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-PINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060A-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:26 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:29:26 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-purple.html>: HTTP status code is not handled or not allowed 2025-11-01 14:29:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-direct-thermal-fanfold-labels-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:29:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:29:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-PINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:29:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060A-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:06 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:30:06 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 14:30:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9313 pages (at 4 pages/min), scraped 4328 items (at 0 items/min) 2025-11-01 14:30:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-direct-thermal-fanfold-labels-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-0-5-sheet-labels-blockout-55.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:19 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:30:19 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-dark-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 14:30:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-50785-sheet-labels-blockout-73.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-PINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060A-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:30:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:30:47 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:30:47 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 14:30:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:30:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:30:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9322 pages (at 9 pages/min), scraped 4328 items (at 0 items/min) 2025-11-01 14:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-FLGREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:31:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:31:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:46 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:31:46 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-ORANGE>: HTTP status code is not handled or not allowed 2025-11-01 14:31:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:31:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-1875-x-1-sheet-labels-blockout-365.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:50 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:31:50 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 14:31:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-FLGREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:07 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:32:08 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-polypropylene.html>: HTTP status code is not handled or not allowed 2025-11-01 14:32:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9330 pages (at 8 pages/min), scraped 4330 items (at 2 items/min) 2025-11-01 14:32:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:32:14 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:32:14 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-10-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 14:32:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-9375-x-3-9375-sheet-labels-blockout-410.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:22 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:32:22 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blockout-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 14:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-blockout-585.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-1875-x-1-sheet-labels-blockout-365.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:32:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:32:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:32:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:39 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:32:39 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 14:32:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-FLGREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-1-125-sheet-labels-blockout-865.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-9375-x-3-9375-sheet-labels-blockout-410.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:56 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:32:57 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 14:32:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:32:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-blockout-585.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-1875-x-1-sheet-labels-blockout-365.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:33:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:33:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-white-matte-litho-77.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9334 pages (at 4 pages/min), scraped 4330 items (at 0 items/min) 2025-11-01 14:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-white-matte-litho-136.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-FLGREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-1-125-sheet-labels-blockout-865.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-9375-x-3-9375-sheet-labels-blockout-410.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-blockout-585.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-1875-x-1-sheet-labels-blockout-365.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-6875-x-2-sheet-labels-white-matte-litho-640.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-white-matte-litho-77.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:33:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:33:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:33:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:33:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-RED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9348 pages (at 14 pages/min), scraped 4335 items (at 5 items/min) 2025-11-01 14:34:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:34:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-CHARTREUSE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:34:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:34:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:34:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:34:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:35:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:35:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9386 pages (at 38 pages/min), scraped 4354 items (at 19 items/min) 2025-11-01 14:35:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:35:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:35:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:35:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:35:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:35:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:35:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:36:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9424 pages (at 38 pages/min), scraped 4368 items (at 14 items/min) 2025-11-01 14:36:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:36:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9463 pages (at 39 pages/min), scraped 4387 items (at 19 items/min) 2025-11-01 14:37:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:37:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9502 pages (at 39 pages/min), scraped 4407 items (at 20 items/min) 2025-11-01 14:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:38:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:38:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9541 pages (at 39 pages/min), scraped 4427 items (at 20 items/min) 2025-11-01 14:39:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:39:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:39:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:39:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:40:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9581 pages (at 40 pages/min), scraped 4444 items (at 17 items/min) 2025-11-01 14:40:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:17 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:40:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:40:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:40:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9620 pages (at 39 pages/min), scraped 4465 items (at 21 items/min) 2025-11-01 14:41:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:41:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:41:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:42:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:42:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9660 pages (at 40 pages/min), scraped 4486 items (at 21 items/min) 2025-11-01 14:42:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:17 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:42:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:42:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:42:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:43:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9699 pages (at 39 pages/min), scraped 4503 items (at 17 items/min) 2025-11-01 14:43:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:43:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:43:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:44:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9739 pages (at 40 pages/min), scraped 4522 items (at 19 items/min) 2025-11-01 14:44:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:44:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:44:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:44:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:44:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:45:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9778 pages (at 39 pages/min), scraped 4540 items (at 18 items/min) 2025-11-01 14:45:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:45:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:45:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:45:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG3030APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:45:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-PUR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9819 pages (at 41 pages/min), scraped 4560 items (at 20 items/min) 2025-11-01 14:46:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:46:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S904AA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:46:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:46:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4060-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S909BB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:46:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT3010AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:46:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4065AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S904AA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-fanfold-labels-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:46:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:47:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S909BB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9842 pages (at 23 pages/min), scraped 4572 items (at 12 items/min) 2025-11-01 14:47:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-3-core-nonperforated-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT3010AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:47:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-top-coated-direct-thermal-fanfold-labels-freezer.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4065AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S904AA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-fanfold-labels-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S909BB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-3-core-nonperforated-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-top-coated-direct-thermal-fanfold-labels-freezer.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4065AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:47:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:48:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S904AA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-fanfold-labels-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9842 pages (at 0 pages/min), scraped 4572 items (at 0 items/min) 2025-11-01 14:48:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:48:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S909BB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:48:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-top-coated-direct-thermal-fanfold-labels-freezer.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-fanfold-labels-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:48:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:48:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S904AA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-fanfold-labels-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:48:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S909BB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-0-75-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:48:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-top-coated-direct-thermal-fanfold-labels-freezer.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:49:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9842 pages (at 0 pages/min), scraped 4572 items (at 0 items/min) 2025-11-01 14:49:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S904AA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-fanfold-labels-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:49:35 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 14:49:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:35 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-red.html>: HTTP status code is not handled or not allowed 2025-11-01 14:49:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:42 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 14:49:42 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-purple-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 14:49:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:49:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:49:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:53 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:49:53 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S904AA>: HTTP status code is not handled or not allowed 2025-11-01 14:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:49:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:03 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:50:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:03 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-chartreuse.html>: HTTP status code is not handled or not allowed 2025-11-01 14:50:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9864 pages (at 22 pages/min), scraped 4577 items (at 5 items/min) 2025-11-01 14:50:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:50:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-1-sheet-labels-blockout-16.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-4-75-sheet-labels-blockout-26.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:50:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-1-sheet-labels-blockout-16.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-4-75-sheet-labels-blockout-26.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9870 pages (at 6 pages/min), scraped 4579 items (at 2 items/min) 2025-11-01 14:51:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:51:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-4-75-sheet-labels-blockout-26.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:51:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:51:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:51:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9870 pages (at 0 pages/min), scraped 4579 items (at 0 items/min) 2025-11-01 14:52:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:18 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 14:52:18 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 14:52:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-75-x-0-75-sheet-labels-blockout-184.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-4-75-sheet-labels-blockout-26.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:52:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:52:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4080A-TOP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:53:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9886 pages (at 16 pages/min), scraped 4586 items (at 7 items/min) 2025-11-01 14:53:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:53:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:12 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 14:53:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:12 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-gray-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 14:53:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4040AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4065AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:53:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4080AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:53:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:53:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9928 pages (at 42 pages/min), scraped 4606 items (at 20 items/min) 2025-11-01 14:54:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:54:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:54:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:54:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:54:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:55:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 9966 pages (at 38 pages/min), scraped 4625 items (at 19 items/min) 2025-11-01 14:55:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:55:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:55:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:55:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10005 pages (at 39 pages/min), scraped 4641 items (at 16 items/min) 2025-11-01 14:56:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:56:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:56:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10045 pages (at 40 pages/min), scraped 4662 items (at 21 items/min) 2025-11-01 14:57:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:57:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:57:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:57:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:57:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10084 pages (at 39 pages/min), scraped 4681 items (at 19 items/min) 2025-11-01 14:58:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:58:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:59:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10123 pages (at 39 pages/min), scraped 4701 items (at 20 items/min) 2025-11-01 14:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:59:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 14:59:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 14:59:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:00:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10161 pages (at 38 pages/min), scraped 4717 items (at 16 items/min) 2025-11-01 15:00:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:00:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:00:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:00:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10200 pages (at 39 pages/min), scraped 4734 items (at 17 items/min) 2025-11-01 15:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:01:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:01:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:01:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:01:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10239 pages (at 39 pages/min), scraped 4756 items (at 22 items/min) 2025-11-01 15:02:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:02:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:02:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:02:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:02:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10280 pages (at 41 pages/min), scraped 4776 items (at 20 items/min) 2025-11-01 15:03:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:03:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG3040APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG3060APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:03:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG3580APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=Z*0600*1476*W*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10318 pages (at 38 pages/min), scraped 4792 items (at 16 items/min) 2025-11-01 15:04:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4090AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:04:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTPF4030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:04:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S905GG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S914EE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-poly.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10353 pages (at 35 pages/min), scraped 4810 items (at 18 items/min) 2025-11-01 15:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:05:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-poly.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:05:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10382 pages (at 29 pages/min), scraped 4824 items (at 14 items/min) 2025-11-01 15:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-poly.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:06:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-poly.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:07:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10390 pages (at 8 pages/min), scraped 4831 items (at 7 items/min) 2025-11-01 15:07:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:07:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:07:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:07:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:07:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:07:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:01 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:08:01 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-purple.html>: HTTP status code is not handled or not allowed 2025-11-01 15:08:03 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:08:03 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-chartreuse-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 15:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-5-x-1-sheet-labels-blockout-182.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10392 pages (at 2 pages/min), scraped 4831 items (at 0 items/min) 2025-11-01 15:08:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:11 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:08:12 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-red.html>: HTTP status code is not handled or not allowed 2025-11-01 15:08:13 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:08:13 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-chartreuse.html>: HTTP status code is not handled or not allowed 2025-11-01 15:08:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:08:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:27 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:08:28 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 15:08:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-5-x-2-sheet-labels-blockout-207.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-5-x-1-sheet-labels-blockout-182.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:47 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:08:47 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core.html>: HTTP status code is not handled or not allowed 2025-11-01 15:08:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-5-sheet-labels-blockout-255.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-3-666-sheet-labels-blockout-320.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:08:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10397 pages (at 5 pages/min), scraped 4831 items (at 0 items/min) 2025-11-01 15:09:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:13 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 15:09:14 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 15:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-RED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040A-RED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4010AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:36 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 15:09:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:36 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-yellow-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 15:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-blockout-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:45 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:09:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:46 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core.html>: HTTP status code is not handled or not allowed 2025-11-01 15:09:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:09:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:09:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-875-x-2-6875-sheet-labels-blockout-650.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:09:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-4-875-sheet-labels-blockout-861.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:10:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10429 pages (at 32 pages/min), scraped 4843 items (at 12 items/min) 2025-11-01 15:10:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:10:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:10:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:10:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:10:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT6080AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:10:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:10:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:11:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10461 pages (at 32 pages/min), scraped 4856 items (at 13 items/min) 2025-11-01 15:11:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:11:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:12:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:12:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10502 pages (at 41 pages/min), scraped 4877 items (at 21 items/min) 2025-11-01 15:12:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:12:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:12:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:12:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:12:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10543 pages (at 41 pages/min), scraped 4895 items (at 18 items/min) 2025-11-01 15:13:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:13:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:13:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:13:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:13:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:14:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:14:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10583 pages (at 40 pages/min), scraped 4917 items (at 22 items/min) 2025-11-01 15:14:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:14:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:14:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10621 pages (at 38 pages/min), scraped 4936 items (at 19 items/min) 2025-11-01 15:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10661 pages (at 40 pages/min), scraped 4953 items (at 17 items/min) 2025-11-01 15:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:16:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10699 pages (at 38 pages/min), scraped 4971 items (at 18 items/min) 2025-11-01 15:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:17:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:17:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:18:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10738 pages (at 39 pages/min), scraped 4991 items (at 20 items/min) 2025-11-01 15:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:18:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:40 [py.warnings] (PID: 101) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 15:18:41 [scrapy.extensions.feedexport] (PID: 101) INFO: Stored bq feed (1000 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:5 2025-11-01 15:18:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:18:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10775 pages (at 37 pages/min), scraped 5009 items (at 18 items/min) 2025-11-01 15:19:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:19:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:19:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10814 pages (at 39 pages/min), scraped 5027 items (at 18 items/min) 2025-11-01 15:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:20:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10854 pages (at 40 pages/min), scraped 5046 items (at 19 items/min) 2025-11-01 15:21:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:21:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:21:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTP4040AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG2515APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:21:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:21:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:22:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S907GG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10895 pages (at 41 pages/min), scraped 5066 items (at 20 items/min) 2025-11-01 15:22:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4040AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT1510AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060A already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDPF4010AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4040A-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:22:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060A already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10925 pages (at 30 pages/min), scraped 5081 items (at 15 items/min) 2025-11-01 15:23:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDPF4010AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4040A-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060A already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:23:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:23:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDPF4010AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4040A-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10926 pages (at 1 pages/min), scraped 5082 items (at 1 items/min) 2025-11-01 15:24:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:24:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:24:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDPF4010AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:24:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:24:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4040A-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:24:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:25:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10926 pages (at 0 pages/min), scraped 5082 items (at 0 items/min) 2025-11-01 15:25:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4040A-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:19 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:25:19 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-gray-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 15:25:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:25:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:25:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:25:42 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:25:42 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 15:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:25:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4040A-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:25:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:00 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:26:00 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-FLORANGE>: HTTP status code is not handled or not allowed 2025-11-01 15:26:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:05 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:26:05 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-blockout.html>: HTTP status code is not handled or not allowed 2025-11-01 15:26:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10930 pages (at 4 pages/min), scraped 5082 items (at 0 items/min) 2025-11-01 15:26:09 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:26:09 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-POLY>: HTTP status code is not handled or not allowed 2025-11-01 15:26:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:26:11 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:26:11 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-5-x-1-5-thermal-transfer-labels-3-core-hot-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 15:26:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:17 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:26:17 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 15:26:18 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:26:19 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-polypropylene.html>: HTTP status code is not handled or not allowed 2025-11-01 15:26:20 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:26:20 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-FLORANGE-XL>: HTTP status code is not handled or not allowed 2025-11-01 15:26:22 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:26:22 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-FLGREEN>: HTTP status code is not handled or not allowed 2025-11-01 15:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:27 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:26:27 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-red-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 15:26:29 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:26:29 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-RED>: HTTP status code is not handled or not allowed 2025-11-01 15:26:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:32 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:26:32 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4040A-TOP>: HTTP status code is not handled or not allowed 2025-11-01 15:26:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:35 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:26:35 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-FLGREEN>: HTTP status code is not handled or not allowed 2025-11-01 15:26:37 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:26:37 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-GREEN-XL>: HTTP status code is not handled or not allowed 2025-11-01 15:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:44 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:26:45 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-pink-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 15:26:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-75-x-2-75-sheet-labels-blockout-80.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-sheet-labels-blockout-99.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:26:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-11-sheet-labels-blockout-129.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10942 pages (at 12 pages/min), scraped 5082 items (at 0 items/min) 2025-11-01 15:27:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:27:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:27:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:27:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:27:20 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:27:20 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-brown-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 15:27:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:27:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-sheet-labels-blockout-99.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-11-sheet-labels-blockout-129.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-nonperforated-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:54 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:27:54 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 15:27:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:27:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-512-x-1-378-sheet-labels-blockout-853.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:27:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10944 pages (at 2 pages/min), scraped 5082 items (at 0 items/min) 2025-11-01 15:28:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-sheet-labels-blockout-99.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:13 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:28:13 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-green.html>: HTTP status code is not handled or not allowed 2025-11-01 15:28:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:22 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 15:28:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:22 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-green-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 15:28:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:28:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GRAPE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4010AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:28:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:28:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:29:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:29:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 10978 pages (at 34 pages/min), scraped 5091 items (at 9 items/min) 2025-11-01 15:29:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:29:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:55 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 15:29:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:29:56 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 15:29:57 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 15:29:58 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 15:29:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:30:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:30:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11012 pages (at 34 pages/min), scraped 5103 items (at 12 items/min) 2025-11-01 15:30:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:30:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:30:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:30:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:30:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:30:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11051 pages (at 39 pages/min), scraped 5121 items (at 18 items/min) 2025-11-01 15:31:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:31:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:31:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11093 pages (at 42 pages/min), scraped 5146 items (at 25 items/min) 2025-11-01 15:32:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:32:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:32:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:32:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11132 pages (at 39 pages/min), scraped 5167 items (at 21 items/min) 2025-11-01 15:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:33:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:33:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:34:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11173 pages (at 41 pages/min), scraped 5184 items (at 17 items/min) 2025-11-01 15:34:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:34:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:34:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:34:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:35:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:35:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:35:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11213 pages (at 40 pages/min), scraped 5201 items (at 17 items/min) 2025-11-01 15:35:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:35:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:35:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:35:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:35:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:35:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:35:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:35:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:35:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:35:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:35:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:35:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:36:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11251 pages (at 38 pages/min), scraped 5221 items (at 20 items/min) 2025-11-01 15:36:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:36:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:36:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:36:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:37:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:37:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:37:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11291 pages (at 40 pages/min), scraped 5243 items (at 22 items/min) 2025-11-01 15:37:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:37:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:37:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:37:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:38:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11331 pages (at 40 pages/min), scraped 5262 items (at 19 items/min) 2025-11-01 15:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:38:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:38:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:38:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG3020APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:38:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG1510APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:39:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:39:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-LTGN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:39:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:39:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11370 pages (at 39 pages/min), scraped 5276 items (at 14 items/min) 2025-11-01 15:39:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:39:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:39:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:39:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:39:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:39:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:39:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:39:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:39:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4020 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:39:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:39:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT2010AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:39:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:39:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:39:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4040AP-2X already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060AP-G already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDF4060AP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11409 pages (at 39 pages/min), scraped 5295 items (at 19 items/min) 2025-11-01 15:40:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4040-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3050A-TOP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-CHARTREUSE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:40:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4030A-TOP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:40:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:41:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11437 pages (at 28 pages/min), scraped 5313 items (at 18 items/min) 2025-11-01 15:41:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:41:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-FLGREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-POLY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:41:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11468 pages (at 31 pages/min), scraped 5327 items (at 14 items/min) 2025-11-01 15:42:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:42:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4065AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:42:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:42:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:57 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 15:42:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:42:57 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blockout.html>: HTTP status code is not handled or not allowed 2025-11-01 15:43:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11498 pages (at 30 pages/min), scraped 5339 items (at 12 items/min) 2025-11-01 15:43:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:20 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 15:43:21 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-purple.html>: HTTP status code is not handled or not allowed 2025-11-01 15:43:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:48 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 15:43:48 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-green.html>: HTTP status code is not handled or not allowed 2025-11-01 15:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:43:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11532 pages (at 34 pages/min), scraped 5354 items (at 15 items/min) 2025-11-01 15:44:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:15 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 15:44:15 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-fluorescent-green.html>: HTTP status code is not handled or not allowed 2025-11-01 15:44:18 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 15:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:18 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-grape.html>: HTTP status code is not handled or not allowed 2025-11-01 15:44:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:29 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 15:44:29 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 15:44:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:44:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:44:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:44:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:44:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-3-666-sheet-labels-crystal-clear-320.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11564 pages (at 32 pages/min), scraped 5367 items (at 13 items/min) 2025-11-01 15:45:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-375-x-1-4375-sheet-labels-crystal-clear-520.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:13 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:45:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-0-946-sheet-labels-crystal-clear-840.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-64-x-0-42-sheet-labels-red-620.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-125-x-4-875-sheet-labels-gray-59.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:45:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:45:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11595 pages (at 31 pages/min), scraped 5384 items (at 17 items/min) 2025-11-01 15:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:46:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:46:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11635 pages (at 40 pages/min), scraped 5402 items (at 18 items/min) 2025-11-01 15:47:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:47:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:47:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:47:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11677 pages (at 42 pages/min), scraped 5423 items (at 21 items/min) 2025-11-01 15:48:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:48:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:48:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:48:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:49:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11715 pages (at 38 pages/min), scraped 5442 items (at 19 items/min) 2025-11-01 15:49:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:49:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-sheet-labels-pastel-orange-91.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-25-sheet-labels-pastel-orange-165.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-25-x-10-75-sheet-labels-pastel-orange-485.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:49:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:49:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:49:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:49:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:50:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11753 pages (at 38 pages/min), scraped 5460 items (at 18 items/min) 2025-11-01 15:50:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:50:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:50:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-6-x-1-77-sheet-labels-fluorescent-red-430.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:50:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:51:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:51:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11789 pages (at 36 pages/min), scraped 5476 items (at 16 items/min) 2025-11-01 15:51:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:51:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:51:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:51:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:51:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:51:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:51:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:51:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:51:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:51:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:51:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:51:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:51:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:51:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:51:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11828 pages (at 39 pages/min), scraped 5494 items (at 18 items/min) 2025-11-01 15:52:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:52:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:52:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11869 pages (at 41 pages/min), scraped 5516 items (at 22 items/min) 2025-11-01 15:53:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:53:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:53:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:53:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:54:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11908 pages (at 39 pages/min), scraped 5534 items (at 18 items/min) 2025-11-01 15:54:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:54:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:54:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-HOTPNK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:54:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S380WX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:54:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S636WX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S926AA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11946 pages (at 38 pages/min), scraped 5549 items (at 15 items/min) 2025-11-01 15:55:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:55:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-PINK-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4010AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:55:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSD0006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0006 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:55:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:55:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4080AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSC0003 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:55:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4013A-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 11983 pages (at 37 pages/min), scraped 5568 items (at 19 items/min) 2025-11-01 15:56:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:17 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:56:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:56:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:56:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:56:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:56:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:56:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:57:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12002 pages (at 19 pages/min), scraped 5579 items (at 11 items/min) 2025-11-01 15:57:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:57:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:57:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12002 pages (at 0 pages/min), scraped 5579 items (at 0 items/min) 2025-11-01 15:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:36 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:58:36 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-dark-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 15:58:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:47 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:58:47 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-hot-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 15:58:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-0-75-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:58:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12004 pages (at 2 pages/min), scraped 5579 items (at 0 items/min) 2025-11-01 15:59:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 15:59:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:17 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:59:17 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-dark-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 15:59:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:21 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 15:59:21 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-yellow.html>: HTTP status code is not handled or not allowed 2025-11-01 15:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3030AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:46 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 15:59:46 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-chartreuse.html>: HTTP status code is not handled or not allowed 2025-11-01 15:59:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 15:59:57 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 15:59:57 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-chartreuse.html>: HTTP status code is not handled or not allowed 2025-11-01 16:00:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:00:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:05 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 16:00:06 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-yellow.html>: HTTP status code is not handled or not allowed 2025-11-01 16:00:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:00:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:00:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12027 pages (at 23 pages/min), scraped 5588 items (at 9 items/min) 2025-11-01 16:00:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:19 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 16:00:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:19 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-grape-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:00:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:23 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 16:00:23 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-dark-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 16:00:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-direct-thermal-labels-3-core-continuous-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-59375-x-1-3-sheet-labels-blockout-76.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/10-6-x-1-18-sheet-labels-blockout-82.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-blockout-136.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-x-1-sheet-labels-blockout-187.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:45 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:00:45 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-fluorescent-red.html>: HTTP status code is not handled or not allowed 2025-11-01 16:00:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:00:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-10-875-sheet-labels-blockout-340.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:00:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-direct-thermal-labels-3-core-continuous-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-59375-x-1-3-sheet-labels-blockout-76.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/10-6-x-1-18-sheet-labels-blockout-82.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12036 pages (at 9 pages/min), scraped 5591 items (at 3 items/min) 2025-11-01 16:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-blockout-136.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-x-1-sheet-labels-blockout-187.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-10-875-sheet-labels-blockout-340.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-direct-thermal-labels-3-core-continuous-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-59375-x-1-3-sheet-labels-blockout-76.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:01:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-blockout-136.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-57-x-4-57-sheet-labels-blockout-545.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-x-1-sheet-labels-blockout-187.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:01:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:01:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-10-875-sheet-labels-blockout-340.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12036 pages (at 0 pages/min), scraped 5591 items (at 0 items/min) 2025-11-01 16:02:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-direct-thermal-labels-3-core-continuous-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:02:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-blockout-136.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-57-x-4-57-sheet-labels-blockout-545.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-x-1-sheet-labels-blockout-187.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-0625-x-2-1875-sheet-labels-blockout-755.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:38 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:02:39 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-brown-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:02:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-10-875-sheet-labels-blockout-340.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-direct-thermal-labels-3-core-continuous-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:02:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:00 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 16:03:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:00 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-dark-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 16:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:03:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12047 pages (at 11 pages/min), scraped 5593 items (at 2 items/min) 2025-11-01 16:03:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:03:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4000-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:03:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:03:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:04:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12086 pages (at 39 pages/min), scraped 5612 items (at 19 items/min) 2025-11-01 16:04:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:04:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:04:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:04:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:04:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:05:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12128 pages (at 42 pages/min), scraped 5632 items (at 20 items/min) 2025-11-01 16:05:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:05:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:05:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:05:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:05:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:05:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12168 pages (at 40 pages/min), scraped 5649 items (at 17 items/min) 2025-11-01 16:06:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:06:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:06:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:06:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:06:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12206 pages (at 38 pages/min), scraped 5666 items (at 17 items/min) 2025-11-01 16:07:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:07:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:07:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:08:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12247 pages (at 41 pages/min), scraped 5688 items (at 22 items/min) 2025-11-01 16:08:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:08:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12286 pages (at 39 pages/min), scraped 5708 items (at 20 items/min) 2025-11-01 16:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:09:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:09:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:10:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12325 pages (at 39 pages/min), scraped 5727 items (at 19 items/min) 2025-11-01 16:10:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:10:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:10:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:10:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:10:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:10:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12363 pages (at 38 pages/min), scraped 5746 items (at 19 items/min) 2025-11-01 16:11:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:11:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:11:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:12:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12405 pages (at 42 pages/min), scraped 5766 items (at 20 items/min) 2025-11-01 16:12:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:12:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:12:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:12:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:12:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12443 pages (at 38 pages/min), scraped 5782 items (at 16 items/min) 2025-11-01 16:13:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=Z*0512*1476*W*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:13:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-FLGRN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTF4020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:13:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTP4010AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=800*190*200*110*H already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12484 pages (at 41 pages/min), scraped 5803 items (at 21 items/min) 2025-11-01 16:14:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:14:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:14:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:14:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-FLGRN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=Z*0512*1476*W*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTF4020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:14:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:14:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-integrated-form-sheet-labels-s929bb.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3050AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-FLGRN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:14:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:14:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12504 pages (at 20 pages/min), scraped 5812 items (at 9 items/min) 2025-11-01 16:15:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-inch-x-3-3125-inch-thank-you-and-safe-handling-instructions-mettler-toledo-labels.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S929BB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-direct-thermal-labels-3-core-non-perf.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3050AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-orange-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-inch-x-3-3125-inch-thank-you-and-safe-handling-instructions-mettler-toledo-labels.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S929BB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-FLGRN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:15:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-0-75-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:15:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4050AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12507 pages (at 3 pages/min), scraped 5813 items (at 1 items/min) 2025-11-01 16:16:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-orange-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-inch-x-3-3125-inch-thank-you-and-safe-handling-instructions-mettler-toledo-labels.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-FLGRN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-direct-thermal-labels-0-75-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:16:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:16:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-5-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-orange-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-1-core-freezer.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-inch-x-3-3125-inch-thank-you-and-safe-handling-instructions-mettler-toledo-labels.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-FLGRN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:17:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-fanfold-labels-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12507 pages (at 0 pages/min), scraped 5813 items (at 0 items/min) 2025-11-01 16:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:17:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:17:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-direct-thermal-labels-0-75-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:17:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-GRAPE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4060AP-ORG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDF4060AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:17:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:48 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 16:17:48 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-yellow-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:17:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4060A-TOP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:17:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-FLGREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12535 pages (at 28 pages/min), scraped 5824 items (at 11 items/min) 2025-11-01 16:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3030AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4030AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:18:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:49 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 16:18:49 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-brown.html>: HTTP status code is not handled or not allowed 2025-11-01 16:18:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4030AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:18:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:19:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12554 pages (at 19 pages/min), scraped 5835 items (at 11 items/min) 2025-11-01 16:19:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-sheet-labels-blockout-45.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-33-sheet-labels-blockout-51.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-blockout-70.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4030AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-sheet-labels-blockout-45.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-33-sheet-labels-blockout-51.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-blockout-70.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:19:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:20:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4030AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12554 pages (at 0 pages/min), scraped 5835 items (at 0 items/min) 2025-11-01 16:20:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:14 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:20:14 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-blockout.html>: HTTP status code is not handled or not allowed 2025-11-01 16:20:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:20:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:20:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-sheet-labels-blockout-45.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-33-sheet-labels-blockout-51.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-blockout-70.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4030AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-blockout-100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-polypropylene-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:20:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:20:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:21:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-blockout-142.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-sheet-labels-blockout-45.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-33-sheet-labels-blockout-51.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:08 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:21:08 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-orange-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:21:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12556 pages (at 2 pages/min), scraped 5835 items (at 0 items/min) 2025-11-01 16:21:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-blockout-70.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:21:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4030AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT5030-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:26 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:21:26 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 16:21:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-blockout-100.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:21:34 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 16:21:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:34 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-polypropylene-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:21:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4010AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:21:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:05 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 16:22:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:22:05 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-yellow.html>: HTTP status code is not handled or not allowed 2025-11-01 16:22:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4015AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12582 pages (at 26 pages/min), scraped 5840 items (at 5 items/min) 2025-11-01 16:22:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:22:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:22:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4090AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-9-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4090AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:22:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:22:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4090AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12615 pages (at 33 pages/min), scraped 5857 items (at 17 items/min) 2025-11-01 16:23:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:23:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT6040AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:23:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:23:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4090AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:23:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-sheet-labels-photo-gloss-885.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12654 pages (at 39 pages/min), scraped 5877 items (at 20 items/min) 2025-11-01 16:24:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-0-5-sheet-labels-white-mid-gloss-55.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:24:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:24:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:24:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:25:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:25:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12692 pages (at 38 pages/min), scraped 5896 items (at 19 items/min) 2025-11-01 16:25:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:25:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:26:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12733 pages (at 41 pages/min), scraped 5915 items (at 19 items/min) 2025-11-01 16:26:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:26:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:26:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:27:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12773 pages (at 40 pages/min), scraped 5935 items (at 20 items/min) 2025-11-01 16:27:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:27:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:27:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:28:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12812 pages (at 39 pages/min), scraped 5955 items (at 20 items/min) 2025-11-01 16:28:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:17 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:28:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:28:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:29:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12854 pages (at 42 pages/min), scraped 5973 items (at 18 items/min) 2025-11-01 16:29:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:29:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:29:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:29:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:30:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12893 pages (at 39 pages/min), scraped 5992 items (at 19 items/min) 2025-11-01 16:30:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:30:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:33 [py.warnings] (PID: 101) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 16:30:34 [scrapy.extensions.feedexport] (PID: 101) INFO: Stored bq feed (1000 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:6 2025-11-01 16:30:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:30:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:30:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:31:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12931 pages (at 38 pages/min), scraped 6010 items (at 18 items/min) 2025-11-01 16:31:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:31:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:31:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:31:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:32:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 12970 pages (at 39 pages/min), scraped 6028 items (at 18 items/min) 2025-11-01 16:32:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:32:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S826FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:33:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13007 pages (at 37 pages/min), scraped 6045 items (at 17 items/min) 2025-11-01 16:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:33:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:34:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13048 pages (at 41 pages/min), scraped 6066 items (at 21 items/min) 2025-11-01 16:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:34:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S28SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S695SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG6060APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=D*0600*1345*W*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:34:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=9310 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=9304 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13086 pages (at 38 pages/min), scraped 6086 items (at 20 items/min) 2025-11-01 16:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:35:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:35:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-FLGREEN-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-FLORANGE-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-GRAY-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:35:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:02 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4060AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13121 pages (at 35 pages/min), scraped 6103 items (at 17 items/min) 2025-11-01 16:36:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSD0005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4020AP-2X already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4012-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-SENSOR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:36:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:37:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13151 pages (at 30 pages/min), scraped 6118 items (at 15 items/min) 2025-11-01 16:37:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:37:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-direct-thermal-labels-3-core-nonperforated-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:37:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:37:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-direct-thermal-labels-3-core-nonperforated-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-2-thermal-transfer-labels-1-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13151 pages (at 0 pages/min), scraped 6118 items (at 0 items/min) 2025-11-01 16:38:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-direct-thermal-labels-3-core-nonperforated-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:38:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:56 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:38:56 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-red.html>: HTTP status code is not handled or not allowed 2025-11-01 16:38:57 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:38:57 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 16:38:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:38:59 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:39:00 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:39:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13154 pages (at 3 pages/min), scraped 6118 items (at 0 items/min) 2025-11-01 16:39:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:11 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:39:11 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-dark-blue-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:39:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:39:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-direct-thermal-labels-1-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:39:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:44 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:39:44 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-1-thermal-transfer-labels-1-core-polypropylene.html>: HTTP status code is not handled or not allowed 2025-11-01 16:39:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-sheet-labels-blockout-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-0-68-sheet-labels-blockout-18.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-direct-thermal-labels-1-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-0-75-sheet-labels-blockout-29.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:39:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13156 pages (at 2 pages/min), scraped 6118 items (at 0 items/min) 2025-11-01 16:40:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:19 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:19 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core.html>: HTTP status code is not handled or not allowed 2025-11-01 16:40:21 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:21 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-PINK>: HTTP status code is not handled or not allowed 2025-11-01 16:40:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:24 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:24 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-nonperforated-fluorescent-green.html>: HTTP status code is not handled or not allowed 2025-11-01 16:40:25 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:25 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-FLRED>: HTTP status code is not handled or not allowed 2025-11-01 16:40:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-sheet-labels-blockout-10.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-0-68-sheet-labels-blockout-18.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-direct-thermal-labels-1-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-0-75-sheet-labels-blockout-29.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:35 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:35 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-5-thermal-transfer-fanfold-labels-brown.html>: HTTP status code is not handled or not allowed 2025-11-01 16:40:37 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:37 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 16:40:38 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:38 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-YELLOW>: HTTP status code is not handled or not allowed 2025-11-01 16:40:39 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:39 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-7-thermal-transfer-fanfold-labels-dark-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 16:40:41 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:41 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-GRAPE>: HTTP status code is not handled or not allowed 2025-11-01 16:40:42 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:42 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-HOTPINK>: HTTP status code is not handled or not allowed 2025-11-01 16:40:44 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:44 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-DARKBLUE>: HTTP status code is not handled or not allowed 2025-11-01 16:40:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:40:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:40:45 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:45 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-4-thermal-transfer-labels-3-core-dark-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 16:40:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:40:46 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:47 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-brown.html>: HTTP status code is not handled or not allowed 2025-11-01 16:40:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:51 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:51 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-5-thermal-transfer-labels-1-core-polypropylene.html>: HTTP status code is not handled or not allowed 2025-11-01 16:40:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:54 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:40:54 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-nonperforated-blue-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:40:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:40:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:40:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:41:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:04 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:41:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-0-68-sheet-labels-blockout-18.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:07 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:41:07 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 16:41:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13172 pages (at 16 pages/min), scraped 6118 items (at 0 items/min) 2025-11-01 16:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-direct-thermal-labels-1-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-0-75-sheet-labels-blockout-29.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-3-66-sheet-labels-blockout-290.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-75-x-3-sheet-labels-blockout-310.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-25-x-3-25-sheet-labels-blockout-862.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-sheet-labels-white-matte-litho-204.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-5-sheet-labels-white-matte-litho-206.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-9-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-9-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-0-68-sheet-labels-blockout-18.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:41:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-0-75-sheet-labels-blockout-29.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-3-66-sheet-labels-blockout-290.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:41:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-75-x-3-sheet-labels-blockout-310.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-25-x-3-25-sheet-labels-blockout-862.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:42:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13172 pages (at 0 pages/min), scraped 6118 items (at 0 items/min) 2025-11-01 16:42:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:42:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-5-sheet-labels-white-matte-litho-206.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-9-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-9-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-0-68-sheet-labels-blockout-18.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-0-75-sheet-labels-blockout-29.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:33 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-75-x-3-sheet-labels-blockout-310.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-25-x-3-25-sheet-labels-blockout-862.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:40 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:42:40 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-blockout.html>: HTTP status code is not handled or not allowed 2025-11-01 16:42:42 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:42:42 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/6-x-4-thermal-transfer-fanfold-labels-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 16:42:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-8-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-5-sheet-labels-white-matte-litho-206.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:42:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-9-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:42:56 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:42:57 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/7-x-0-68-sheet-labels-blockout-18.html>: HTTP status code is not handled or not allowed 2025-11-01 16:42:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:02 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:43:02 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/7-x-0-75-sheet-labels-blockout-29.html>: HTTP status code is not handled or not allowed 2025-11-01 16:43:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13176 pages (at 4 pages/min), scraped 6118 items (at 0 items/min) 2025-11-01 16:43:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-9-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-75-x-3-sheet-labels-blockout-310.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-25-x-3-25-sheet-labels-blockout-862.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-656-x-4-656-sheet-labels-photo-gloss-97.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-photo-gloss-106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-8-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-5-sheet-labels-white-matte-litho-206.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-2-75-sheet-labels-photo-gloss-170.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-9-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-875-x-2-6875-sheet-labels-photo-gloss-650.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-0625-x-2-1875-sheet-labels-photo-gloss-755.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-9-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-75-x-3-sheet-labels-blockout-310.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-25-x-3-25-sheet-labels-blockout-862.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-656-x-4-656-sheet-labels-photo-gloss-97.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-photo-gloss-106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-8-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-5-sheet-labels-white-matte-litho-206.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-2-75-sheet-labels-photo-gloss-170.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:02 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:02 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-9-thermal-transfer-labels-3-core-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:05 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:05 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-yellow-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:07 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:07 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13179 pages (at 3 pages/min), scraped 6118 items (at 0 items/min) 2025-11-01 16:44:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-875-x-2-6875-sheet-labels-photo-gloss-650.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-0625-x-2-1875-sheet-labels-photo-gloss-755.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:16 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:16 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:44:18 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:18 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-yellow.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:19 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:20 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-dark-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:21 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:21 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-5-thermal-transfer-labels-3-core-yellow.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-9-thermal-transfer-labels-3-core-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:24 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:24 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-75-x-3-sheet-labels-blockout-310.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:26 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:26 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-yellow.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:27 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:27 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-25-x-3-25-sheet-labels-blockout-862.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:28 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:28 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-3-core-hot-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-656-x-4-656-sheet-labels-photo-gloss-97.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-4-sheet-labels-photo-gloss-106.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-8-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:36 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:36 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/5-x-5-sheet-labels-white-matte-litho-206.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-2-75-sheet-labels-photo-gloss-170.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-1494-x-3-3072-sheet-labels-photo-gloss-828.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:40 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:40 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-9-thermal-transfer-labels-3-core-chartreuse.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-1-125-sheet-labels-photo-gloss-865.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-1-1-sheet-labels-photo-gloss-871.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:45 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:45 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/6-x-4-thermal-transfer-labels-3-core-polypropylene.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-875-x-2-6875-sheet-labels-photo-gloss-650.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:48 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:44:48 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 16:44:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-0625-x-2-1875-sheet-labels-photo-gloss-755.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-3-sheet-labels-white-mid-gloss-213.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:44:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:44:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-125-x-1-75-sheet-labels-white-premium-high-gloss-36.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S193GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13196 pages (at 17 pages/min), scraped 6118 items (at 0 items/min) 2025-11-01 16:45:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:45:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S828GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S330GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:45:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13238 pages (at 42 pages/min), scraped 6138 items (at 20 items/min) 2025-11-01 16:46:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:46:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:18 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:46:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S545CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:46:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S9CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:46:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13280 pages (at 42 pages/min), scraped 6155 items (at 17 items/min) 2025-11-01 16:47:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:47:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S595VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:47:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13321 pages (at 41 pages/min), scraped 6173 items (at 18 items/min) 2025-11-01 16:48:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:48:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S625PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:48:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:48:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:48:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:49:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13360 pages (at 39 pages/min), scraped 6194 items (at 21 items/min) 2025-11-01 16:49:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:49:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:49:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S103PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:49:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:49:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:50:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:50:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:50:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13402 pages (at 42 pages/min), scraped 6217 items (at 23 items/min) 2025-11-01 16:50:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:50:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:50:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:50:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:51:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13442 pages (at 40 pages/min), scraped 6235 items (at 18 items/min) 2025-11-01 16:51:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:51:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S891FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:52 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:51:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:51:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S112FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13481 pages (at 39 pages/min), scraped 6253 items (at 18 items/min) 2025-11-01 16:52:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S520FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:52:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:52:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:52:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:53:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13519 pages (at 38 pages/min), scraped 6270 items (at 17 items/min) 2025-11-01 16:53:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S875FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:53:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S99FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:53:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:53:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:53:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:53:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:53:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S610GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13559 pages (at 40 pages/min), scraped 6290 items (at 20 items/min) 2025-11-01 16:54:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S620SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:54:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:54:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-DKB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:54:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:54:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=9306 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13598 pages (at 39 pages/min), scraped 6310 items (at 20 items/min) 2025-11-01 16:55:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WST0009 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:55:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4025-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FDT4013-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:55:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13624 pages (at 26 pages/min), scraped 6323 items (at 13 items/min) 2025-11-01 16:56:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:56:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:56:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:57:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13636 pages (at 12 pages/min), scraped 6330 items (at 7 items/min) 2025-11-01 16:57:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-1-core-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:57:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:57:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-brown-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-gray-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-1-core-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:57:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13636 pages (at 0 pages/min), scraped 6330 items (at 0 items/min) 2025-11-01 16:58:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:10 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:58:10 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-purple-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:58:12 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:58:12 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 16:58:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:58:16 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:58:16 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-fluorescent-red-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:58:18 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:58:18 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-brown-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:58:19 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:58:19 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-nonperforated-gray-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:58:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:58:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-1-core-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:24 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:58:24 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core.html>: HTTP status code is not handled or not allowed 2025-11-01 16:58:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:29 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:58:29 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 16:58:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:58:31 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:58:31 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-fluorescent-green.html>: HTTP status code is not handled or not allowed 2025-11-01 16:58:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-625-x-4-625-sheet-labels-blockout-75.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:58:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-1-core-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13644 pages (at 8 pages/min), scraped 6330 items (at 0 items/min) 2025-11-01 16:59:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:59:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:20 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 16:59:20 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:59:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-3-core-nonperforated-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 16:59:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-625-x-4-625-sheet-labels-blockout-75.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:39 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:59:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-1-core-top-coated-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:39 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-hot-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 16:59:40 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:59:40 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-10-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:59:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-3-core-nonperforated-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:48 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:59:48 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-FLRED>: HTTP status code is not handled or not allowed 2025-11-01 16:59:49 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:59:49 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-fluorescent-orange-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 16:59:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:52 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 16:59:53 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-fluorescent-green.html>: HTTP status code is not handled or not allowed 2025-11-01 16:59:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 16:59:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:00:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-1875-x-1-5-sheet-labels-blockout-760.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:03 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:00:03 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-yellow.html>: HTTP status code is not handled or not allowed 2025-11-01 17:00:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-625-x-4-625-sheet-labels-blockout-75.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13655 pages (at 11 pages/min), scraped 6332 items (at 2 items/min) 2025-11-01 17:00:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-polypropylene.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:12 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:00:12 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-DARKBLUE-XL>: HTTP status code is not handled or not allowed 2025-11-01 17:00:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:15 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:00:15 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-direct-thermal-labels-1-core-top-coated-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 17:00:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:00:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-3-core-nonperforated-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-0-5-sheet-labels-blockout-883.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-2-sheet-labels-white-matte-litho-202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-10-875-sheet-labels-white-matte-litho-340.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-purple-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-1875-x-1-5-sheet-labels-blockout-760.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-625-x-4-625-sheet-labels-blockout-75.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:49 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:00:50 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/5-x-3-thermal-transfer-fanfold-labels-polypropylene.html>: HTTP status code is not handled or not allowed 2025-11-01 17:00:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-white-matte-litho-355.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:00:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-0-75-sheet-labels-white-matte-litho-515.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:00:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-3-core-nonperforated-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:02 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:01:03 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-x-6-thermal-transfer-labels-3-core-fluorescent-red.html>: HTTP status code is not handled or not allowed 2025-11-01 17:01:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-0-5-sheet-labels-blockout-883.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-2-sheet-labels-white-matte-litho-202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13659 pages (at 4 pages/min), scraped 6332 items (at 0 items/min) 2025-11-01 17:01:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-10-875-sheet-labels-white-matte-litho-340.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-grape-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:12 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:01:12 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-3-core-nonperforated-purple-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 17:01:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-1875-x-1-5-sheet-labels-blockout-760.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-625-x-4-625-sheet-labels-blockout-75.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-x-3-thermal-transfer-labels-3-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-11-sheet-labels-white-matte-litho-355.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-white-matte-litho-585.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-0-75-sheet-labels-white-matte-litho-515.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-direct-thermal-labels-3-core-nonperforated-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-9-sheet-labels-white-matte-litho-615.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:01:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-2-sheet-labels-white-matte-litho-202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-375-x-10-875-sheet-labels-white-matte-litho-340.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:01:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-375-x-1-25-sheet-labels-white-matte-litho-775.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-1875-x-1-5-sheet-labels-blockout-760.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3510AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3535AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:01:58 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:01:58 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-625-x-4-625-sheet-labels-blockout-75.html>: HTTP status code is not handled or not allowed 2025-11-01 17:02:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:02:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:02:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13665 pages (at 6 pages/min), scraped 6332 items (at 0 items/min) 2025-11-01 17:02:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:11 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 17:02:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4080A-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:11 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-fluorescent-green.html>: HTTP status code is not handled or not allowed 2025-11-01 17:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:14 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 17:02:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2013, in _inlineCallbacks result = context.run( ^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 68, in process_request return (yield download_func(request, spider)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ scrapy.core.downloader.handlers.http11.TunnelError: Could not open CONNECT tunnel with proxy brd.superproxy.io:22225 [{'status': 502, 'reason': b'Proxy Error: server_error Error creating ext request: TUN_ERR: Destination host connect timeout'}] 2025-11-01 17:02:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4010AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-CHARTREUSE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-9375-x-0-5-sheet-labels-photo-gloss-217.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:02:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4080A-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4010AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:02:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:02:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-CHARTREUSE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13696 pages (at 31 pages/min), scraped 6344 items (at 12 items/min) 2025-11-01 17:03:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:03:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-75-x-1-75-sheet-labels-photo-gloss-860.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-25-x-0-75-sheet-labels-photo-gloss-869.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-5-x-0-75-sheet-labels-white-mid-gloss-864.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4010AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-0-75-sheet-labels-white-premium-high-gloss-189.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:03:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:03:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:03:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:03:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S440CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13729 pages (at 33 pages/min), scraped 6358 items (at 14 items/min) 2025-11-01 17:04:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:04:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:04:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:04:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:04:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:04:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:04:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13769 pages (at 40 pages/min), scraped 6379 items (at 21 items/min) 2025-11-01 17:05:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S204TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:05:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:05:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:05:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:05:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:05:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S310WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:05:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S59VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S86VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13810 pages (at 41 pages/min), scraped 6399 items (at 20 items/min) 2025-11-01 17:06:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S421VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:06:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:06:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:06:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:06:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S715PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:07:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13850 pages (at 40 pages/min), scraped 6419 items (at 20 items/min) 2025-11-01 17:07:10 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:07:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:07:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:34 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:07:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S827PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:07:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S340PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13890 pages (at 40 pages/min), scraped 6436 items (at 17 items/min) 2025-11-01 17:08:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:08:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:08:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:08:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S884PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:09:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:09:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13929 pages (at 39 pages/min), scraped 6456 items (at 20 items/min) 2025-11-01 17:09:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S320BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:09:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:10:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:03 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:10:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 13969 pages (at 40 pages/min), scraped 6476 items (at 20 items/min) 2025-11-01 17:10:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S104FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S755FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:10:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S41FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:10:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:10:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S116FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:11:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14010 pages (at 41 pages/min), scraped 6496 items (at 20 items/min) 2025-11-01 17:11:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S720FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:11:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:12:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14050 pages (at 40 pages/min), scraped 6516 items (at 20 items/min) 2025-11-01 17:12:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S862SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=I*0410*500*W*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=Z*0433*1476*X*BX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-ORG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTP4020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTPF4060AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=800*190*200*1000*H already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:51 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:12:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:12:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:12:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S901AA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:01 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:13:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S917DD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4060-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14089 pages (at 39 pages/min), scraped 6534 items (at 18 items/min) 2025-11-01 17:13:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSB0002 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:13:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:13:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0005 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4060AP-2X already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4010AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4080AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:13:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4040AP-SENSOR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4020AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4020A already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:13:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4030-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2515AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14123 pages (at 34 pages/min), scraped 6550 items (at 16 items/min) 2025-11-01 17:14:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:11 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:14:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4030AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLGREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:14:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:14:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:14:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3020AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:14:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-1-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14149 pages (at 26 pages/min), scraped 6566 items (at 16 items/min) 2025-11-01 17:15:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:08 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:15:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-656-x-4-656-sheet-labels-blockout-35.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT3020AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3040AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-1-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-25-x-1-25-sheet-labels-blockout-92.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:49 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:15:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-656-x-4-656-sheet-labels-blockout-35.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:15:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:15:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:16:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:16:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14149 pages (at 0 pages/min), scraped 6566 items (at 0 items/min) 2025-11-01 17:16:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-1-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-25-x-1-25-sheet-labels-blockout-92.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:24 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:16:24 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-hot-pink.html>: HTTP status code is not handled or not allowed 2025-11-01 17:16:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-656-x-4-656-sheet-labels-blockout-35.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:35 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:16:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:38 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:16:38 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 17:16:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:16:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:43 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:16:44 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-25-x-1-25-thermal-transfer-labels-1-core-dark-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 17:16:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:16:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-gray.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-1-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-25-x-1-25-sheet-labels-blockout-92.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-33-x-3-33-sheet-labels-blockout-145.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-5-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-dark-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-656-x-4-656-sheet-labels-blockout-35.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14152 pages (at 3 pages/min), scraped 6566 items (at 0 items/min) 2025-11-01 17:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-12-thermal-transfer-labels-3-core-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-green-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-5-x-0-5-sheet-labels-blockout-208.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blockout.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-4375-x-5-4531-sheet-labels-blockout-306.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-5-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:22 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:17:22 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-thermal-transfer-labels-1-core-gray.html>: HTTP status code is not handled or not allowed 2025-11-01 17:17:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-thermal-transfer-labels-3-core-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-2-thermal-transfer-labels-3-core-nonperforated-red-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-direct-thermal-labels-1-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-25-x-1-25-sheet-labels-blockout-92.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-fanfold-labels-chartreuse.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-BROWN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3010AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-33-x-3-33-sheet-labels-blockout-145.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:41 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 17:17:41 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-2-thermal-transfer-labels-1-core-blue.html>: HTTP status code is not handled or not allowed 2025-11-01 17:17:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-chartreuse-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4013AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:17:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:17:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:17:56 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 17:17:56 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-blockout.html>: HTTP status code is not handled or not allowed 2025-11-01 17:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-RED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4010AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14171 pages (at 19 pages/min), scraped 6568 items (at 2 items/min) 2025-11-01 17:18:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4040AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S710OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:24 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 17:18:24 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-6-5-thermal-transfer-labels-1-core-chartreuse-xl.html>: HTTP status code is not handled or not allowed 2025-11-01 17:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:50 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 500 Internal Server Error 2025-11-01 17:18:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:50 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <500 https://www.smithcorona.com/4-x-8-thermal-transfer-labels-1-core-red.html>: HTTP status code is not handled or not allowed 2025-11-01 17:18:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4065AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:18:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S88ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:18:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14208 pages (at 37 pages/min), scraped 6588 items (at 20 items/min) 2025-11-01 17:19:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S196ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4090AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4090AP-BLOCKOUT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4090AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:19:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S470GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S877MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:19:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14249 pages (at 41 pages/min), scraped 6609 items (at 21 items/min) 2025-11-01 17:20:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S90GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S730GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S435GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:23 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:20:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:20:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S60CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S82CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S87SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:20:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:56 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:20:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:20:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:21:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S92GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14288 pages (at 39 pages/min), scraped 6624 items (at 15 items/min) 2025-11-01 17:21:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:21:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:21:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:21:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S570GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:21:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:21:58 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:21:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14327 pages (at 39 pages/min), scraped 6643 items (at 19 items/min) 2025-11-01 17:22:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:22:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:22:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:22:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:22:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S460VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:22:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S113PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:22:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:22:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14365 pages (at 38 pages/min), scraped 6661 items (at 18 items/min) 2025-11-01 17:23:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S865PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S42PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S201PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S203PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:23:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S823PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S838PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14402 pages (at 37 pages/min), scraped 6682 items (at 21 items/min) 2025-11-01 17:24:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S325PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S550PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:24:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:24:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:26 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:24:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S505PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S890PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:24:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:24:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S47PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:24:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S75PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:24:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S110PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14444 pages (at 42 pages/min), scraped 6701 items (at 19 items/min) 2025-11-01 17:25:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S840PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S874PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:25:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:25:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S180FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S210FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:25:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:26:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14483 pages (at 39 pages/min), scraped 6717 items (at 16 items/min) 2025-11-01 17:26:14 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:26:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:26:17 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:26:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S160FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S20FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:42 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:26:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:26:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:26:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S125FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S155FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S345FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14523 pages (at 40 pages/min), scraped 6733 items (at 16 items/min) 2025-11-01 17:27:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S525FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S760FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S816FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:27:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S853FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:38 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:27:39 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:27:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S280FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S355FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:27:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:27:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:28:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14562 pages (at 39 pages/min), scraped 6755 items (at 22 items/min) 2025-11-01 17:28:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S78FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S831FB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S44GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:28:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:24 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:28:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S665GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S705GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S863GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:28:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:28:55 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:28:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14603 pages (at 41 pages/min), scraped 6773 items (at 18 items/min) 2025-11-01 17:29:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG4050APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG3535APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT5030AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG6080APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-B already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S35SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-1-25-sheet-labels-240.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:44 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:29:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S100SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:29:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S405WX already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:30:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S902AA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S915CC already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:30:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14637 pages (at 34 pages/min), scraped 6791 items (at 18 items/min) 2025-11-01 17:30:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-BROWN-XS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:30:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:30:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:30:28 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:30:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2010AP-DARKBLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDF4060AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:45 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:30:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-GRAPE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4060AP-G already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:30:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14669 pages (at 32 pages/min), scraped 6805 items (at 14 items/min) 2025-11-01 17:31:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:31:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-PINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:31:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4060AP-G already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-blue-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:47 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:31:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:31:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:31:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-1-direct-thermal-labels-3-core-top-coated.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4050A-TOP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:31:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4013-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14687 pages (at 18 pages/min), scraped 6815 items (at 10 items/min) 2025-11-01 17:32:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4060AP-G already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-1-sheet-labels-blockout-54.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-blockout-71.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-35-sheet-labels-blockout-103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:32:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:32:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-25-x-1-625-sheet-labels-blockout-114.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RDT4050A-TOP-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:41 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:32:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4060AP-G already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060A-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-5-x-1-sheet-labels-blockout-54.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-green.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-blockout-71.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:32:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-35-sheet-labels-blockout-103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-25-x-1-625-sheet-labels-blockout-114.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:33:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14687 pages (at 0 pages/min), scraped 6815 items (at 0 items/min) 2025-11-01 17:33:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4040A-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4060AP-G already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:33:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:25 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:33:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:28 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:33:28 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-6-thermal-transfer-labels-1-core-fluorescent-green.html>: HTTP status code is not handled or not allowed 2025-11-01 17:33:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-blockout-71.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:31 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:33:31 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-x-1-thermal-transfer-labels-1-core-fluorescent-orange.html>: HTTP status code is not handled or not allowed 2025-11-01 17:33:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-35-sheet-labels-blockout-103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-25-x-1-625-sheet-labels-blockout-114.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-5-x-1-75-sheet-labels-blockout-183.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:33:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4060AP-G already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:33:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-brown.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-sheet-labels-blockout-415.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-blockout-71.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14689 pages (at 2 pages/min), scraped 6815 items (at 0 items/min) 2025-11-01 17:34:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-66-x-1-375-sheet-labels-blockout-565.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-orange.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-35-sheet-labels-blockout-103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-25-x-1-625-sheet-labels-blockout-114.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-5-x-1-75-sheet-labels-blockout-183.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:27 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:34:27 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-3-thermal-transfer-labels-1-core-brown.html>: HTTP status code is not handled or not allowed 2025-11-01 17:34:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-purple.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-375-sheet-labels-blockout-839.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:35 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:34:35 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4060AP-G>: HTTP status code is not handled or not allowed 2025-11-01 17:34:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-0-thermal-transfer-labels-3-core-continuous-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:41 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:34:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-sheet-labels-blockout-415.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:41 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-4-thermal-transfer-labels-1-core-brown.html>: HTTP status code is not handled or not allowed 2025-11-01 17:34:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-blockout-71.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-66-x-1-375-sheet-labels-blockout-565.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-yellow.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:46 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:34:47 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020AP-BROWN>: HTTP status code is not handled or not allowed 2025-11-01 17:34:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-nonperforated-hot-pink.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-35-sheet-labels-blockout-103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-5-x-1-75-sheet-labels-blockout-183.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GRAY-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:58 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:34:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:34:58 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-HOTPINK>: HTTP status code is not handled or not allowed 2025-11-01 17:34:59 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:34:59 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-5-x-3-5-thermal-transfer-labels-3-core-purple.html>: HTTP status code is not handled or not allowed 2025-11-01 17:35:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-10-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-375-sheet-labels-blockout-839.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:03 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:35:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:03 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-5-x-8-thermal-transfer-labels-3-core-red.html>: HTTP status code is not handled or not allowed 2025-11-01 17:35:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-1-sheet-labels-white-matte-litho-15.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14700 pages (at 11 pages/min), scraped 6816 items (at 1 items/min) 2025-11-01 17:35:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-sheet-labels-blockout-415.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:15 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:35:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:15 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/4-x-1-5-sheet-labels-blockout-71.html>: HTTP status code is not handled or not allowed 2025-11-01 17:35:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-66-x-1-375-sheet-labels-blockout-565.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-35-sheet-labels-white-matte-litho-103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:22 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:35:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:22 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/1-5-x-1-35-sheet-labels-blockout-103.html>: HTTP status code is not handled or not allowed 2025-11-01 17:35:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-5-x-1-75-sheet-labels-blockout-183.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:31 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:35:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S190ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:31 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-GRAY-XL>: HTTP status code is not handled or not allowed 2025-11-01 17:35:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-8-thermal-transfer-labels-3-core-grape.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S880OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-375-sheet-labels-blockout-839.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-1-sheet-labels-white-matte-litho-15.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S213ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-sheet-labels-blockout-415.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4090AP-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:35:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-35-sheet-labels-white-matte-litho-103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-66-x-1-375-sheet-labels-blockout-565.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5625-x-1-75-sheet-labels-white-matte-litho-440.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/0-5-x-1-75-sheet-labels-blockout-183.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14716 pages (at 16 pages/min), scraped 6818 items (at 2 items/min) 2025-11-01 17:36:09 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:36:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S114OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:36:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:36:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-375-sheet-labels-blockout-839.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:22 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:36:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-1-sheet-labels-white-matte-litho-15.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:30 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-13-thermal-transfer-labels-3-core-nonperforated-hot-pink-xl.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:31 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:36:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S350ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-35-sheet-labels-white-matte-litho-103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:36 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:36:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-PINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5625-x-1-75-sheet-labels-white-matte-litho-440.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-66-x-1-375-sheet-labels-blockout-565.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-875-x-2-6875-sheet-labels-white-matte-litho-650.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-8-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-9-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:36:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-1-sheet-labels-white-matte-litho-15.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5-x-2-sheet-labels-photo-gloss-202.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:07 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-HOTPINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14731 pages (at 15 pages/min), scraped 6824 items (at 6 items/min) 2025-11-01 17:37:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4000-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5625-x-1-75-sheet-labels-white-matte-litho-440.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-35-sheet-labels-white-matte-litho-103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:15 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:37:16 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:37:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:16 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/2-66-x-1-375-sheet-labels-blockout-565.html>: HTTP status code is not handled or not allowed 2025-11-01 17:37:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-0-5-sheet-labels-photo-gloss-883.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-8-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-1-5-sheet-labels-white-mid-gloss-71.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-x-2-sheet-labels-white-mid-gloss-77.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:43 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:37:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S415OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5625-x-1-75-sheet-labels-white-matte-litho-440.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-5-x-1-35-sheet-labels-white-matte-litho-103.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:37:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-6-thermal-transfer-labels-3-core-fluorescent-red.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-x-8-thermal-transfer-labels-3-core-dark-blue.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S615MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14748 pages (at 17 pages/min), scraped 6831 items (at 7 items/min) 2025-11-01 17:38:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/6-78-x-4-75-sheet-labels-white-mid-gloss-848.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:19 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:38:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-25-x-0-75-sheet-labels-white-mid-gloss-869.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-5-x-4-125-sheet-labels-white-premium-high-gloss-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-5625-x-1-75-sheet-labels-white-matte-litho-440.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:29 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:38:29 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/1-5-x-1-35-sheet-labels-white-matte-litho-103.html>: HTTP status code is not handled or not allowed 2025-11-01 17:38:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:38:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S108GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-sheet-labels-white-premium-high-gloss-130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:38:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6080AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:46 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:38:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:50 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:38:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:56 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/1-25-x-0-75-sheet-labels-white-mid-gloss-869.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-0-875-sheet-labels-white-premium-high-gloss-180.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:38:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S140GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:02 [scrapy.downloadermiddlewares.retry] (PID: 101) ERROR: Gave up retrying (failed 6 times): 403 Forbidden 2025-11-01 17:39:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/5-5-x-4-125-sheet-labels-white-premium-high-gloss-9.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:02 [scrapy.spidermiddlewares.httperror] (PID: 101) INFO: Ignoring response <403 https://www.smithcorona.com/3-5625-x-1-75-sheet-labels-white-matte-litho-440.html>: HTTP status code is not handled or not allowed 2025-11-01 17:39:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-66-x-1-375-sheet-labels-white-premium-high-gloss-565.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:39:05 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:39:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14766 pages (at 18 pages/min), scraped 6840 items (at 9 items/min) 2025-11-01 17:39:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-sheet-labels-white-premium-high-gloss-130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-75-x-2-5-sheet-labels-white-premium-high-gloss-867.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-sheet-labels-white-premium-high-gloss-876.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-875-x-3-875-sheet-labels-clear-matte-395.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-0-75-sheet-labels-crystal-clear-29.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-625-x-0-875-sheet-labels-white-premium-high-gloss-180.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-x-3-33-sheet-labels-crystal-clear-96.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:39:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-11-sheet-labels-crystal-clear-129.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-64-x-4-64-sheet-labels-crystal-clear-115.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-sheet-labels-white-premium-high-gloss-130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S192CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-sheet-labels-white-premium-high-gloss-876.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-75-x-2-5-sheet-labels-white-premium-high-gloss-867.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:39:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-0-75-sheet-labels-crystal-clear-29.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-25-x-10-75-sheet-labels-crystal-clear-485.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:40:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14781 pages (at 15 pages/min), scraped 6846 items (at 6 items/min) 2025-11-01 17:40:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-65-x-4-65-sheet-labels-crystal-clear-490.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-25-x-11-sheet-labels-crystal-clear-129.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-64-x-4-64-sheet-labels-crystal-clear-115.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-sheet-labels-white-premium-high-gloss-130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-sheet-labels-white-premium-high-gloss-876.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:27 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:40:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S165CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S395CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/8-25-x-10-75-sheet-labels-crystal-clear-485.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S860CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/7-x-4-875-sheet-labels-crystal-clear-861.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S96CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S565GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S869MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S10SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/4-65-x-4-65-sheet-labels-crystal-clear-490.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S129CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:40:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S115CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/3-x-5-sheet-labels-white-premium-high-gloss-876.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S198CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/2-83-x-1-sheet-labels-white-premium-high-gloss-130.html already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S70SR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:41:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14797 pages (at 16 pages/min), scraped 6852 items (at 6 items/min) 2025-11-01 17:41:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:12 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:41:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S848MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:21 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S485CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S861CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S490CCL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S876GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S130GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S136GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:57 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:41:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829GR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:41:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:42:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:42:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S55TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:08 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14831 pages (at 34 pages/min), scraped 6871 items (at 19 items/min) 2025-11-01 17:42:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S410TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S690TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:16 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S822TU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:17 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S181WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:21 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:42:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S200WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S260WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S475WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:39 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871WPL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S117VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:45 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S142VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S185VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:42:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:00 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S495VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S650VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:43:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14870 pages (at 39 pages/min), scraped 6890 items (at 19 items/min) 2025-11-01 17:43:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885VD already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S54PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S430PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:23 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:43:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:33 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871PP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S208PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S175PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:48 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:43:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:54 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:43:54 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:58 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S675PU already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:43:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S202PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S209PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S207PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:06 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:44:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14911 pages (at 41 pages/min), scraped 6909 items (at 19 items/min) 2025-11-01 17:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885PB already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S255PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S360PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867PG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S51PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S95PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:32 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:44:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S780PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S839PY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:46 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S127PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S135PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:44:53 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:44:55 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:01 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S817PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14948 pages (at 37 pages/min), scraped 6927 items (at 18 items/min) 2025-11-01 17:45:11 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S832PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S871PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:14 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S883PO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S97PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S18PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S183PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:27 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:45:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S240PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S785PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:38 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S285PT already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S71BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:48 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S81BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S187BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:45:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S189BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S205BK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 14985 pages (at 37 pages/min), scraped 6945 items (at 18 items/min) 2025-11-01 17:46:09 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S26FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S76FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:12 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S106FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S390FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S420FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S515FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:25 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S500FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:29 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:46:31 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S864FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:32 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680FR already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:37 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S194FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:40 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:43 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S170FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:44 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S215FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S220FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:50 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S290FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:53 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S306FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:57 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S365FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:46:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:47:00 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:47:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S530FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:04 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770FP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 15025 pages (at 40 pages/min), scraped 6965 items (at 20 items/min) 2025-11-01 17:47:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S199FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:13 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S217FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:15 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S605FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:20 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:47:22 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S835FG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S46FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:30 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:47:34 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S89FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:35 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S735FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:40 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:47:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S745FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:42 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S824FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S885FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S867FY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:52 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S138FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:59 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S212FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:47:59 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:48:02 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S630FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:03 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S872FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:05 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:06 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878FO already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:07 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:48:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 15065 pages (at 40 pages/min), scraped 6982 items (at 17 items/min) 2025-11-01 17:48:10 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S45GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:16 [scrapy.core.scraper] (PID: 101) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 2017, in _inlineCallbacks result = context.run(gen.send, result) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 57, in process_request method(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scraping_utils/middlewares/downloaders/spoofer.py", line 101, in process_request generated_headers = self.generator.generate() ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 159, in generate generated: Dict[str, str] = self._get_headers( ^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 267, in _get_headers return self.order_headers({**generated_sample, **request_dependent_headers}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/browserforge/headers/generator.py", line 357, in order_headers raise ValueError("Failed to find User-Agent in generated response") ValueError: Failed to find User-Agent in generated response 2025-11-01 17:48:18 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:19 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S214GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:20 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S195GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:24 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S535GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:26 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S700GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:28 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S810GF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:29 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S109SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:36 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S137SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:37 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S191SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:41 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S590SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:47 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S655SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:49 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:51 [HeadersSpooferDownloaderMiddleware] (PID: 101) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S770SF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2025-11-01 17:48:57 [py.warnings] (PID: 101) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2025-11-01 17:48:58 [scrapy.extensions.feedexport] (PID: 101) INFO: Stored bq feed (1000 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:7 2025-11-01 17:49:08 [scrapy.extensions.logstats] (PID: 101) INFO: Crawled 15104 pages (at 39 pages/min), scraped 7006 items (at 24 items/min) 2025-11-01 17:49:12 [scrapy.core.engine] (PID: 101) INFO: Closing spider (finished) 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] ------------------------------ MONITORS ------------------------------ 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] Extracted Items Monitor/test_stat_monitor... FAIL 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] Item Validation Monitor/test_stat_monitor... SKIPPED (Unable to find 'spidermon/validation/fields/errors' in job stats.) 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] Error Count Monitor/test_stat_monitor... FAIL 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] Warning Count Monitor/test_stat_monitor... FAIL 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] Finish Reason Monitor/Should have the expected finished reason(s)... OK 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] Unwanted HTTP codes monitor/Should not hit the limit of unwanted http status... FAIL 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] Field Coverage Monitor/test_check_if_field_coverage_rules_are_met... OK 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] Retry Count monitor/Should not hit the limit of requests that reached the maximum retry amount... FAIL 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] Downloader Exceptions monitor/test_stat_monitor... FAIL 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] Successful Requests monitor/Should have at least the minimum number of successful requests... OK 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] Total Requests monitor/Should not hit the total limit of requests... OK 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] ---------------------------------------------------------------------- 2025-11-01 17:49:12 [smith_corona] (PID: 101) ERROR: [Spidermon] ====================================================================== FAIL: Extracted Items Monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 184, in test_stat_monitor assertion_method( AssertionError: Expecting 'item_scraped_count' to be '>=' to '8000.0'. Current value: '7009' 2025-11-01 17:49:12 [smith_corona] (PID: 101) ERROR: [Spidermon] ====================================================================== FAIL: Error Count Monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 184, in test_stat_monitor assertion_method( AssertionError: Expecting 'log_count/ERROR' to be '<=' to '7.0'. Current value: '1993' 2025-11-01 17:49:12 [smith_corona] (PID: 101) ERROR: [Spidermon] ====================================================================== FAIL: Warning Count Monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 184, in test_stat_monitor assertion_method( AssertionError: Expecting 'log_count/WARNING' to be '<=' to '1000.0'. Current value: '11903' 2025-11-01 17:49:12 [smith_corona] (PID: 101) ERROR: [Spidermon] ====================================================================== FAIL: Unwanted HTTP codes monitor/Should not hit the limit of unwanted http status ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/monitors.py", line 236, in test_check_unwanted_http_codes self.assertTrue(count <= max_errors, msg=msg) AssertionError: Found 586 Responses with status code=500 - This exceeds the limit of 100 2025-11-01 17:49:12 [smith_corona] (PID: 101) ERROR: [Spidermon] ====================================================================== FAIL: Retry Count monitor/Should not hit the limit of requests that reached the maximum retry amount ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/monitors.py", line 282, in test_maximum_retries self.assertLessEqual(max_reached, threshold, msg=msg) AssertionError: Too many requests (364) reached the maximum retry amount 2025-11-01 17:49:12 [smith_corona] (PID: 101) ERROR: [Spidermon] ====================================================================== FAIL: Downloader Exceptions monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 184, in test_stat_monitor assertion_method( AssertionError: Expecting 'downloader/exception_count' to be '<=' to '100.0'. Current value: '1630' 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] 11 monitors in 0.006s 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] FAILED (failures=6, skipped=1) 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] -------------------------- FINISHED ACTIONS -------------------------- 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] ---------------------------------------------------------------------- 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] 0 actions in 0.000s 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] OK 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] --------------------------- PASSED ACTIONS --------------------------- 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] ---------------------------------------------------------------------- 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] 0 actions in 0.000s 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] OK 2025-11-01 17:49:12 [smith_corona] (PID: 101) INFO: [Spidermon] --------------------------- FAILED ACTIONS --------------------------- 2025-11-01 17:49:12 [spidermon.contrib.actions.slack] (PID: 101) WARNING: bot cannot finder user in slack org member list - default icon url used 2025-11-01 17:49:13 [smith_corona] (PID: 101) INFO: [Spidermon] CustomTemplateSendSlackMessageSpiderFinished... OK 2025-11-01 17:49:13 [smith_corona] (PID: 101) INFO: [Spidermon] ---------------------------------------------------------------------- 2025-11-01 17:49:13 [smith_corona] (PID: 101) INFO: [Spidermon] 1 action in 0.423s 2025-11-01 17:49:13 [smith_corona] (PID: 101) INFO: [Spidermon] OK 2025-11-01 17:49:13 [scrapy.extensions.feedexport] (PID: 101) INFO: Stored bq feed (9 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:8 2025-11-01 17:49:13 [scrapy.statscollectors] (PID: 101) INFO: Dumping Scrapy stats: {'HeadersSpooferDownloaderMiddleware/spoofed': 20233, 'NotFoundHandlerSpiderMiddleware/HttpError': 363, 'big_query/url': 9002, 'downloader/exception_count': 1630, 'downloader/exception_type_count/builtins.ValueError': 1628, 'downloader/exception_type_count/scrapy.core.downloader.handlers.http11.TunnelError': 1, 'downloader/exception_type_count/twisted.web._newclient.ResponseNeverReceived': 1, 'downloader/request_bytes': 18877720, 'downloader/request_count': 20233, 'downloader/request_method_count/GET': 20233, 'downloader/response_bytes': 851825971, 'downloader/response_count': 20231, 'downloader/response_status_count/200': 14744, 'downloader/response_status_count/301': 1, 'downloader/response_status_count/403': 4900, 'downloader/response_status_count/500': 586, 'dupefilter/filtered': 1, 'elapsed_time_seconds': 30844.419111, 'feedexport/success_count/BigQueryFeedStorage': 8, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2025, 11, 1, 17, 49, 12, 695251, tzinfo=datetime.timezone.utc), 'httpcompression/response_bytes': 6680638123, 'httpcompression/response_count': 14744, 'httperror/response_ignored_count': 363, 'httperror/response_ignored_status_count/403': 284, 'httperror/response_ignored_status_count/500': 79, 'item_scraped_count': 7009, 'items_per_minute': None, 'log_count/ERROR': 1999, 'log_count/INFO': 927, 'log_count/WARNING': 11904, 'memusage/max': 252489728, 'memusage/startup': 125906944, 'poet/injector/catalog_extraction.pages.smith_corona.SmithCoronaSingleProductPageObject': 7069, 'proxy_manager/ignored/proxy_defined': 5125, 'proxy_manager/processed': 16736, 'request_depth_max': 1, 'response_received_count': 15107, 'responses_per_minute': None, 'retry/count': 5124, 'retry/max_reached': 364, 'retry/reason_count/403 Forbidden': 4616, 'retry/reason_count/500 Internal Server Error': 507, 'retry/reason_count/twisted.web._newclient.ResponseNeverReceived': 1, 'scheduler/dequeued': 21861, 'scheduler/dequeued/memory': 21861, 'scheduler/enqueued': 21861, 'scheduler/enqueued/memory': 21861, 'spidermon/validation/fields': 147189, 'spidermon/validation/items': 7009, 'spidermon/validation/validators': 1, 'spidermon/validation/validators/item/jsonschema': True, 'spidermon_field_coverage/dict/brand': 1.0, 'spidermon_field_coverage/dict/categories': 1.0, 'spidermon_field_coverage/dict/countryOfOrigin': 1.0, 'spidermon_field_coverage/dict/description': 1.0, 'spidermon_field_coverage/dict/imageUrl': 1.0, 'spidermon_field_coverage/dict/inStock': 1.0, 'spidermon_field_coverage/dict/isFreeShipping': 1.0, 'spidermon_field_coverage/dict/leadTime': 1.0, 'spidermon_field_coverage/dict/manufacturer': 1.0, 'spidermon_field_coverage/dict/manufacturerSku': 1.0, 'spidermon_field_coverage/dict/name': 1.0, 'spidermon_field_coverage/dict/packagingIncrement': 1.0, 'spidermon_field_coverage/dict/prices': 1.0, 'spidermon_field_coverage/dict/productStatus': 1.0, 'spidermon_field_coverage/dict/relatedSkus': 1.0, 'spidermon_field_coverage/dict/specifications': 1.0, 'spidermon_field_coverage/dict/supplier': 1.0, 'spidermon_field_coverage/dict/supplierSku': 1.0, 'spidermon_field_coverage/dict/uom': 1.0, 'spidermon_field_coverage/dict/url': 1.0, 'spidermon_field_coverage/dict/weight': 1.0, 'spidermon_item_scraped_count': 7009, 'spidermon_item_scraped_count/dict': 7009, 'spidermon_item_scraped_count/dict/brand': 7009, 'spidermon_item_scraped_count/dict/categories': 7009, 'spidermon_item_scraped_count/dict/countryOfOrigin': 7009, 'spidermon_item_scraped_count/dict/description': 7009, 'spidermon_item_scraped_count/dict/imageUrl': 7009, 'spidermon_item_scraped_count/dict/inStock': 7009, 'spidermon_item_scraped_count/dict/isFreeShipping': 7009, 'spidermon_item_scraped_count/dict/leadTime': 7009, 'spidermon_item_scraped_count/dict/manufacturer': 7009, 'spidermon_item_scraped_count/dict/manufacturerSku': 7009, 'spidermon_item_scraped_count/dict/name': 7009, 'spidermon_item_scraped_count/dict/packagingIncrement': 7009, 'spidermon_item_scraped_count/dict/prices': 7009, 'spidermon_item_scraped_count/dict/productStatus': 7009, 'spidermon_item_scraped_count/dict/relatedSkus': 7009, 'spidermon_item_scraped_count/dict/specifications': 7009, 'spidermon_item_scraped_count/dict/supplier': 7009, 'spidermon_item_scraped_count/dict/supplierSku': 7009, 'spidermon_item_scraped_count/dict/uom': 7009, 'spidermon_item_scraped_count/dict/url': 7009, 'spidermon_item_scraped_count/dict/weight': 7009, 'start_requests/big_query': 9002, 'start_time': datetime.datetime(2025, 11, 1, 9, 15, 8, 276140, tzinfo=datetime.timezone.utc)} 2025-11-01 17:49:13 [scrapy.core.engine] (PID: 101) INFO: Spider closed (finished)