2026-01-24 00:00:12 [scrapy.utils.log] (PID: 14) INFO: Scrapy 2.12.0 started (bot: catalog_extraction) 2026-01-24 00:00:12 [scrapy.utils.log] (PID: 14) INFO: Versions: lxml 5.3.1.0, libxml2 2.12.9, cssselect 1.3.0, parsel 1.10.0, w3lib 2.3.1, Twisted 24.11.0, Python 3.11.13 (main, Jun 10 2025, 23:54:42) [GCC 12.2.0], pyOpenSSL 25.0.0 (OpenSSL 3.4.1 11 Feb 2025), cryptography 44.0.2, Platform Linux-6.9.12-x86_64-with-glibc2.36 2026-01-24 00:00:13 [grainger] (PID: 14) INFO: Starting extraction spider grainger... 2026-01-24 00:00:13 [scrapy.addons] (PID: 14) INFO: Enabled addons: [] 2026-01-24 00:00:13 [py.warnings] (PID: 14) WARNING: /usr/local/lib/python3.11/site-packages/scrapy/utils/request.py:120: ScrapyDeprecationWarning: 'REQUEST_FINGERPRINTER_IMPLEMENTATION' is a deprecated setting. It will be removed in a future version of Scrapy. return cls(crawler) 2026-01-24 00:00:13 [scrapy.extensions.telnet] (PID: 14) INFO: Telnet Password: e6bea2c332056022 2026-01-24 00:00:13 [py.warnings] (PID: 14) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2026-01-24 00:00:15 [scrapy.middleware] (PID: 14) INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.closespider.CloseSpider', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats', 'scrapy_playwright.memusage.ScrapyPlaywrightMemoryUsageExtension', 'spidermon.contrib.scrapy.extensions.Spidermon'] 2026-01-24 00:00:15 [scrapy.crawler] (PID: 14) INFO: Overridden settings: {'BOT_NAME': 'catalog_extraction', 'CONCURRENT_ITEMS': 250, 'CONCURRENT_REQUESTS': 3, 'FEED_EXPORT_ENCODING': 'utf-8', 'HTTPPROXY_ENABLED': False, 'LOG_FILE': '/var/lib/scrapyd/logs/catalog_extraction/grainger/a15de848f8b711f08aaf4200a9fe0102.log', 'LOG_FORMAT': '%(asctime)s [%(name)s] (PID: %(process)d) %(levelname)s: ' '%(message)s', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'catalog_extraction.spiders', 'REQUEST_FINGERPRINTER_CLASS': 'scrapy_poet.ScrapyPoetRequestFingerprinter', 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7', 'SPIDER_MODULES': ['catalog_extraction.spiders'], 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor', 'USER_AGENT': None} 2026-01-24 00:00:15 [scrapy-playwright] (PID: 14) WARNING: Connecting to remote browser, ignoring PLAYWRIGHT_LAUNCH_OPTIONS 2026-01-24 00:00:15 [scrapy-playwright] (PID: 14) WARNING: Connecting to remote browser, ignoring PLAYWRIGHT_LAUNCH_OPTIONS 2026-01-24 00:00:15 [scrapy-playwright] (PID: 14) WARNING: Connecting to remote browser, ignoring PLAYWRIGHT_LAUNCH_OPTIONS 2026-01-24 00:00:15 [scrapy-playwright] (PID: 14) WARNING: Connecting to remote browser, ignoring PLAYWRIGHT_LAUNCH_OPTIONS 2026-01-24 00:00:16 [scrapy_poet.injection] (PID: 14) INFO: Loading providers: [, , , , , , ] 2026-01-24 00:00:16 [scrapy.middleware] (PID: 14) INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scraping_utils.middlewares.downloaders.HeadersSpooferDownloaderMiddleware', 'scrapy_poet.InjectionMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy_poet.DownloaderStatsMiddleware'] 2026-01-24 00:00:16 [NotFoundHandlerSpiderMiddleware] (PID: 14) INFO: NotFoundHandlerSpiderMiddleware running on PRODUCTION environment. 2026-01-24 00:00:16 [scrapy.middleware] (PID: 14) INFO: Enabled spider middlewares: ['catalog_extraction.middlewares.NotFoundHandlerSpiderMiddleware', 'catalog_extraction.middlewares.FixtureSavingMiddleware', 'scrapy_poet.RetryMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2026-01-24 00:00:16 [scrapy.middleware] (PID: 14) INFO: Enabled item pipelines: ['catalog_extraction.pipelines.DuplicatedSKUsFilterPipeline', 'catalog_extraction.pipelines.DiscontinuedProductsAdjustmentPipeline', 'catalog_extraction.pipelines.PriceRoundingPipeline', 'scraping_utils.pipelines.AttachSupplierPipeline', 'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline'] 2026-01-24 00:00:16 [scrapy.core.engine] (PID: 14) INFO: Spider opened 2026-01-24 00:00:16 [scrapy.extensions.closespider] (PID: 14) INFO: Spider will stop when no items are produced after 7200 seconds. 2026-01-24 00:00:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2026-01-24 00:00:16 [scrapy.extensions.telnet] (PID: 14) INFO: Telnet console listening on 127.0.0.1:6023 2026-01-24 00:00:16 [scrapy-playwright] (PID: 14) INFO: Starting download handler 2026-01-24 00:00:16 [scrapy-playwright] (PID: 14) INFO: Starting download handler 2026-01-24 00:01:05 [scrapy-playwright] (PID: 14) INFO: Connecting using CDP: wss://brd-customer-hl_13cda1e4-zone-main_scraping_browser:l9p73ctebkrc@brd.superproxy.io:9222 2026-01-24 00:01:05 [scrapy-playwright] (PID: 14) INFO: Connected using CDP: wss://brd-customer-hl_13cda1e4-zone-main_scraping_browser:l9p73ctebkrc@brd.superproxy.io:9222 2026-01-24 00:01:09 [grainger] (PID: 14) WARNING: Page from CDP session 1 closed in the errback 2026-01-24 00:01:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2026-01-24 00:01:22 [grainger] (PID: 14) INFO: CDP session 0 parsing 2026-01-24 00:01:23 [grainger] (PID: 14) WARNING: URL https://www.grainger.com/product/Cotter-Pin-41JX75 not found in the scheduled URLs 2026-01-24 00:01:26 [grainger] (PID: 14) INFO: CDP session 2 parsing 2026-01-24 00:01:26 [grainger] (PID: 14) WARNING: URL https://www.grainger.com/product/OSG-Square-End-Mill-Carbide-30M161 not found in the scheduled URLs 2026-01-24 00:01:43 [py.warnings] (PID: 14) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2026-01-24 00:01:44 [scrapy.extensions.feedexport] (PID: 14) INFO: Stored bq feed (200 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:1 2026-01-24 00:02:12 [py.warnings] (PID: 14) WARNING: /var/lib/scrapyd/eggs/catalog_extraction/1758126308.egg/catalog_extraction/extensions/bq_feedstorage.py:33: ScrapyDeprecationWarning: scrapy.extensions.feedexport.build_storage() is deprecated, call the builder directly. 2026-01-24 00:02:13 [scrapy.extensions.feedexport] (PID: 14) INFO: Stored bq feed (200 items) in: bq://response-elt.scraper_data.catalog_item_scrape/batch:2 2026-01-24 00:02:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 2 pages/min), scraped 431 items (at 431 items/min) 2026-01-24 00:03:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 128 items/min) 2026-01-24 00:03:22 [grainger] (PID: 14) ERROR: Error fetching through Playwright: Page.goto: net::ERR_TUNNEL_CONNECTION_FAILED at https://www.grainger.com/product/info?productArray=804VT8,61JG27,241CY9,497P65,825VH0,411M31,45CF38,33YW51,61JF86,2VXT8 Call log: - navigating to "https://www.grainger.com/product/info?productArray=804VT8,61JG27,241CY9,497P65,825VH0,411M31,45CF38,33YW51,61JF86,2VXT8", waiting until "load" 2026-01-24 00:03:22 [grainger] (PID: 14) ERROR: Error fetching through Playwright: Page.goto: net::ERR_ABORTED; maybe frame was detached? Call log: - navigating to "https://www.grainger.com/product/info?productArray=38TP48,48HX62,41TF51,61JG68,48KP99,241DH2,38Z656,55ZD63,22JU40,15A756", waiting until "load" 2026-01-24 00:03:22 [grainger] (PID: 14) INFO: CDP session 2 finished 2026-01-24 00:03:22 [grainger] (PID: 14) INFO: CDP session 0 finished 2026-01-24 00:03:24 [grainger] (PID: 14) WARNING: Page from CDP session 4 closed in the errback 2026-01-24 00:03:25 [grainger] (PID: 14) WARNING: Page from CDP session 3 closed in the errback 2026-01-24 00:03:26 [grainger] (PID: 14) WARNING: Page from CDP session 6 closed in the errback 2026-01-24 00:03:27 [grainger] (PID: 14) WARNING: Page from CDP session 9 closed in the errback 2026-01-24 00:03:27 [grainger] (PID: 14) WARNING: Page from CDP session 7 closed in the errback 2026-01-24 00:03:28 [grainger] (PID: 14) WARNING: Page from CDP session 10 closed in the errback 2026-01-24 00:03:28 [grainger] (PID: 14) WARNING: Page from CDP session 8 closed in the errback 2026-01-24 00:03:30 [grainger] (PID: 14) WARNING: Page from CDP session 11 closed in the errback 2026-01-24 00:04:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:05:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:06:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:07:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:08:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:09:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:10:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:11:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:12:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:13:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:14:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:15:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:16:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:17:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:18:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:19:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:20:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:21:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:22:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:23:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:24:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:25:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:26:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:27:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:28:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:29:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:30:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:31:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:32:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:33:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:34:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:35:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:36:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:37:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:38:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:39:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:40:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:41:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:42:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:43:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:44:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:45:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:46:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:47:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:48:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:49:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:50:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:51:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:52:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:53:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:54:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min) 2026-01-24 00:55:16 [scrapy.extensions.logstats] (PID: 14) INFO: Crawled 2 pages (at 0 pages/min), scraped 559 items (at 0 items/min)