2026-01-10 10:15:04 [scrapy.utils.log] (PID: 79) INFO: Scrapy 2.12.0 started (bot: catalog_extraction) 2026-01-10 10:15:04 [scrapy.utils.log] (PID: 79) INFO: Versions: lxml 5.3.1.0, libxml2 2.12.9, cssselect 1.3.0, parsel 1.10.0, w3lib 2.3.1, Twisted 24.11.0, Python 3.11.13 (main, Jun 10 2025, 23:54:42) [GCC 12.2.0], pyOpenSSL 25.0.0 (OpenSSL 3.4.1 11 Feb 2025), cryptography 44.0.2, Platform Linux-6.9.12-x86_64-with-glibc2.36 2026-01-10 10:15:04 [smith_corona] (PID: 79) INFO: Starting extraction spider smith_corona... 2026-01-10 10:15:04 [scrapy.addons] (PID: 79) INFO: Enabled addons: [] 2026-01-10 10:15:04 [py.warnings] (PID: 79) WARNING: /usr/local/lib/python3.11/site-packages/scrapy/utils/request.py:120: ScrapyDeprecationWarning: 'REQUEST_FINGERPRINTER_IMPLEMENTATION' is a deprecated setting. It will be removed in a future version of Scrapy. return cls(crawler) 2026-01-10 10:15:04 [scrapy.extensions.telnet] (PID: 79) INFO: Telnet Password: 462989706f0ca656 2026-01-10 10:15:04 [py.warnings] (PID: 79) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2026-01-10 10:15:05 [scrapy.middleware] (PID: 79) INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.closespider.CloseSpider', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats', 'spidermon.contrib.scrapy.extensions.Spidermon'] 2026-01-10 10:15:05 [scrapy.crawler] (PID: 79) INFO: Overridden settings: {'BOT_NAME': 'catalog_extraction', 'CONCURRENT_ITEMS': 250, 'CONCURRENT_REQUESTS': 24, 'DOWNLOAD_DELAY': 1.25, 'FEED_EXPORT_ENCODING': 'utf-8', 'LOG_FILE': '/var/lib/scrapyd/logs/catalog_extraction/smith_corona/3993f81aee0d11f0a4ea4200a9fe0102.log', 'LOG_FORMAT': '%(asctime)s [%(name)s] (PID: %(process)d) %(levelname)s: ' '%(message)s', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'catalog_extraction.spiders', 'REQUEST_FINGERPRINTER_CLASS': 'scrapy_poet.ScrapyPoetRequestFingerprinter', 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7', 'RETRY_HTTP_CODES': [500, 502, 503, 504, 522, 524, 408, 429, 403], 'RETRY_TIMES': 5, 'SPIDER_MODULES': ['catalog_extraction.spiders'], 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor', 'USER_AGENT': None} 2026-01-10 10:15:05 [scrapy_poet.injection] (PID: 79) INFO: Loading providers: [, , , , , , ] 2026-01-10 10:15:05 [scrapy.middleware] (PID: 79) INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scraping_utils.middlewares.downloaders.ProxyManagerDownloaderMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scraping_utils.middlewares.downloaders.HeadersSpooferDownloaderMiddleware', 'scrapy_poet.InjectionMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy_poet.DownloaderStatsMiddleware'] 2026-01-10 10:15:05 [NotFoundHandlerSpiderMiddleware] (PID: 79) INFO: NotFoundHandlerSpiderMiddleware running on PRODUCTION environment. 2026-01-10 10:15:05 [scrapy.middleware] (PID: 79) INFO: Enabled spider middlewares: ['catalog_extraction.middlewares.NotFoundHandlerSpiderMiddleware', 'catalog_extraction.middlewares.FixtureSavingMiddleware', 'scrapy_poet.RetryMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2026-01-10 10:15:05 [scrapy.middleware] (PID: 79) INFO: Enabled item pipelines: ['catalog_extraction.pipelines.DuplicatedSKUsFilterPipeline', 'catalog_extraction.pipelines.DiscontinuedProductsAdjustmentPipeline', 'catalog_extraction.pipelines.PriceRoundingPipeline', 'scraping_utils.pipelines.AttachSupplierPipeline', 'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline'] 2026-01-10 10:15:05 [scrapy.core.engine] (PID: 79) INFO: Spider opened 2026-01-10 10:15:05 [scrapy.extensions.closespider] (PID: 79) INFO: Spider will stop when no items are produced after 1800 seconds. 2026-01-10 10:15:05 [scrapy.extensions.logstats] (PID: 79) INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2026-01-10 10:15:05 [scrapy.extensions.telnet] (PID: 79) INFO: Telnet console listening on 127.0.0.1:6026 2026-01-10 10:15:08 [ProxyManagerDownloaderMiddleware] (PID: 79) INFO: Using brd-customer-hl_13cda1e4-zone-sharedpool_datacenter_proxy as the default proxy for ProxyManagerDownloaderMiddleware. 2026-01-10 10:15:10 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=R36IJG4025APM already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6090AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:15 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTPF4020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:16 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020AF already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:18 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AP-CHA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4060AF-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:22 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:24 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S29W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:26 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S56W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S295W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:28 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=ETT4060AP-CHARTREUSE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:30 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S775W already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:32 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4060AP-HOTPINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:33 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT3020AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:34 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSB0001 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:35 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=WSH0008 already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:37 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT1510AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:39 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT2010AP-TOP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=ETT2010AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:43 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:45 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=ETT225125AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4020-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:46 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=EDT4080AP-UPS already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4012-BLUE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:15:48 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2015AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:05 [scrapy.extensions.logstats] (PID: 79) INFO: Crawled 38 pages (at 38 pages/min), scraped 13 items (at 13 items/min) 2026-01-10 10:16:23 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT2060AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:29 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=ETT4010AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=HDT4025AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:32 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3020A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:35 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:37 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4040-FLGREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:39 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=ETT4020AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:41 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3050A-PINK-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:42 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3100A-PURPLE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:44 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4065-RED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:45 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4070-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4020A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:47 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=ETT4040AP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:49 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=ETT4040AP-FLORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=ETT4040AP-GREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:53 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT4080-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=ETT4050AP-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:54 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-BLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:57 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4030A-FLGREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:16:59 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-FLGREEN-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:17:00 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=FTT6040-YELLOW already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:17:01 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3060AP-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:17:02 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-GRAPE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:17:04 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-YELLOW-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:17:05 [scrapy.extensions.logstats] (PID: 79) INFO: Crawled 77 pages (at 39 pages/min), scraped 28 items (at 15 items/min) 2026-01-10 10:17:08 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4050A-ORANGE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:17:29 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT3580AP-BROWN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:17:45 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:17:48 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S15OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:17:50 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S84OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:17:54 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-PURPLE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:17:56 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S91OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:17:57 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100A-BLOCKOUT-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:17:58 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4012AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:01 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013AP-GREEN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:03 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4013A-FLRED-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:04 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:05 [scrapy.extensions.logstats] (PID: 79) INFO: Crawled 113 pages (at 36 pages/min), scraped 50 items (at 22 items/min) 2026-01-10 10:18:05 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6040A-DARKBLUE-XL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:07 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S465OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4080A-GRAY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:08 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4025AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:12 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S829OP already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4065AP-ORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:14 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S211ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:15 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-FLRED already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:16 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4090AP-FLORANGE already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:19 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT4100AP-POLY already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:20 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S680ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:21 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=RTT6060AP-PINK already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:24 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S815ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:25 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S866ML already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:18:28 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S11GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:01 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S50GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:02 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S126GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:04 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S133GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:05 [scrapy.extensions.logstats] (PID: 79) INFO: Crawled 155 pages (at 42 pages/min), scraped 76 items (at 26 items/min) 2026-01-10 10:19:08 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S275GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:09 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S182GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:14 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S510GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:14 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S585GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:15 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S870GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S843GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:19 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S878GIN already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:20 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S145MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:20 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S93MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:24 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S206MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S685MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:25 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S184MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S855MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:30 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S640MG already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:32 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S80GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:33 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S77GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:35 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S120GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:38 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S85GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S400GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:39 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S16GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:41 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S868GLA already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:44 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S36CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS. 2026-01-10 10:19:57 [HeadersSpooferDownloaderMiddleware] (PID: 79) WARNING: Request https://www.smithcorona.com/smithcorona_productdatarepository/ProductPage/Index?sku=S73CL already has headers. They will be preserved, but that may lead to fingerprint inconsistency. If the headers are necessary, consider disabling SPOOF_FULL_HEADERS.