2024-09-18 23:27:04 [scrapy.utils.log] INFO: Scrapy 2.11.2 started (bot: catalog_discovery) 2024-09-18 23:27:04 [scrapy.utils.log] INFO: Versions: lxml 5.2.2.0, libxml2 2.12.6, cssselect 1.2.0, parsel 1.9.1, w3lib 2.1.2, Twisted 24.3.0, Python 3.11.9 (main, Aug 13 2024, 01:19:58) [GCC 12.2.0], pyOpenSSL 24.1.0 (OpenSSL 3.2.1 30 Jan 2024), cryptography 42.0.7, Platform Linux-6.4.10-dirty-x86_64-with-glibc2.36 2024-09-18 23:27:04 [scrapy.addons] INFO: Enabled addons: [] 2024-09-18 23:27:04 [scrapy.extensions.telnet] INFO: Telnet Password: 6f560f48fbb70115 2024-09-18 23:27:04 [scrapy.middleware] INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.memusage.MemoryUsage', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats', 'spidermon.contrib.scrapy.extensions.Spidermon'] 2024-09-18 23:27:04 [scrapy.crawler] INFO: Overridden settings: {'BOT_NAME': 'catalog_discovery', 'CONCURRENT_ITEMS': 1000, 'CONCURRENT_REQUESTS': 32, 'FEED_EXPORT_ENCODING': 'utf-8', 'LOG_FILE': '/var/lib/scrapyd/logs/catalog_discovery/global_industrial/80b6f39e761511efb4924200a9fe0102.log', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'catalog_discovery.spiders', 'REQUEST_FINGERPRINTER_CLASS': 'scrapy_poet.ScrapyPoetRequestFingerprinter', 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7', 'RETRY_TIMES': 5, 'SPIDER_MODULES': ['catalog_discovery.spiders'], 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor', 'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:125.0) ' 'Gecko/20100101 Firefox/125.0'} 2024-09-18 23:27:04 [scrapy_poet.injection] INFO: Loading providers: [, , , , , , ] 2024-09-18 23:27:04 [scrapy.middleware] INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware', 'scrapy_poet.InjectionMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy_poet.DownloaderStatsMiddleware'] 2024-09-18 23:27:04 [scrapy.middleware] INFO: Enabled spider middlewares: ['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware', 'scrapy_poet.RetryMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2024-09-18 23:27:04 [scrapy.middleware] INFO: Enabled item pipelines: ['scraping_utils.pipelines.DuplicatesFilterPipeline', 'scraping_utils.pipelines.AttachSupplierPipeline', 'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline'] 2024-09-18 23:27:04 [scrapy.core.engine] INFO: Spider opened 2024-09-18 23:27:04 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2024-09-18 23:27:04 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023 2024-09-18 23:27:10 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:1 2024-09-18 23:27:11 [scrapy.core.downloader.handlers.http11] WARNING: Got data loss in https://www.globalindustrial.com/sitemap-products-12.xml. If you want to process broken responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False -- This message won't be shown in further requests 2024-09-18 23:27:11 [scrapy.core.downloader.handlers.http11] WARNING: Got data loss in https://www.globalindustrial.com/sitemap-products-5.xml. If you want to process broken responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False -- This message won't be shown in further requests 2024-09-18 23:27:11 [scrapy.core.downloader.handlers.http11] WARNING: Got data loss in https://www.globalindustrial.com/sitemap-products-13.xml. If you want to process broken responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False -- This message won't be shown in further requests 2024-09-18 23:27:11 [scrapy.core.downloader.handlers.http11] WARNING: Got data loss in https://www.globalindustrial.com/sitemap-products-3.xml. If you want to process broken responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False -- This message won't be shown in further requests 2024-09-18 23:27:11 [scrapy.core.downloader.handlers.http11] WARNING: Got data loss in https://www.globalindustrial.com/sitemap-products-10.xml. If you want to process broken responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False -- This message won't be shown in further requests 2024-09-18 23:27:11 [scrapy.core.downloader.handlers.http11] WARNING: Got data loss in https://www.globalindustrial.com/sitemap-products-11.xml. If you want to process broken responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False -- This message won't be shown in further requests 2024-09-18 23:27:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:2 2024-09-18 23:27:13 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:3 2024-09-18 23:27:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:4 2024-09-18 23:27:15 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:5 2024-09-18 23:27:16 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:6 2024-09-18 23:27:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:7 2024-09-18 23:27:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:8 2024-09-18 23:27:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:9 2024-09-18 23:27:21 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:10 2024-09-18 23:27:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:11 2024-09-18 23:27:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:12 2024-09-18 23:27:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:13 2024-09-18 23:27:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:14 2024-09-18 23:27:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:15 2024-09-18 23:27:29 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:16 2024-09-18 23:27:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:17 2024-09-18 23:27:31 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/outdoor-permishield-led-wallpack-wphotocell-20w-dark-bronze', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/outdoor-permishield-led-wallpack-wphotocell-20w-dark-bronze', 'sku': None, 'name': None} 2024-09-18 23:27:31 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:18 2024-09-18 23:27:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:19 2024-09-18 23:27:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:20 2024-09-18 23:27:35 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:21 2024-09-18 23:27:36 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:22 2024-09-18 23:27:36 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:23 2024-09-18 23:27:38 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:24 2024-09-18 23:27:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:25 2024-09-18 23:27:40 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:26 2024-09-18 23:27:41 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:27 2024-09-18 23:27:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:28 2024-09-18 23:27:43 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/rubbermaid-top-shelf-for-rubbermaid-cleaning-cart', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/rubbermaid-top-shelf-for-rubbermaid-cleaning-cart', 'sku': None, 'name': None} 2024-09-18 23:27:43 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/rubbermaid-middle-shelf-for-rubbermaid-cleaning-cart', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/rubbermaid-middle-shelf-for-rubbermaid-cleaning-cart', 'sku': None, 'name': None} 2024-09-18 23:27:43 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/rubbermaid-4-swivel-stem-caster-with-insert', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/rubbermaid-4-swivel-stem-caster-with-insert', 'sku': None, 'name': None} 2024-09-18 23:27:43 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/rubbermaid-hardware-kit-for-rubbermaid-utility-carts', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/rubbermaid-hardware-kit-for-rubbermaid-utility-carts', 'sku': None, 'name': None} 2024-09-18 23:27:43 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/rubbermaid-4-swivel-stem-caster-rubbermaid-tilt-truck', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/rubbermaid-4-swivel-stem-caster-rubbermaid-tilt-truck', 'sku': None, 'name': None} 2024-09-18 23:27:43 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/rubbermaid-5-swivel-plate-caster', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/rubbermaid-5-swivel-plate-caster', 'sku': None, 'name': None} 2024-09-18 23:27:43 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/rubbermaid-12-wheel-with-hardware-includes-1-12-wheel-2-washers-1-axle-nut', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/rubbermaid-12-wheel-with-hardware-includes-1-12-wheel-2-washers-1-axle-nut', 'sku': None, 'name': None} 2024-09-18 23:27:43 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/rubbermaid-4-swivel-plate-caster-with-hardware-includes-1-caster-and-4-screws', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/rubbermaid-4-swivel-plate-caster-with-hardware-includes-1-caster-and-4-screws', 'sku': None, 'name': None} 2024-09-18 23:27:43 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/rubbermaid-10-wheel', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/rubbermaid-10-wheel', 'sku': None, 'name': None} 2024-09-18 23:27:43 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/rubbermaid-5-swivel-plate-caster', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/rubbermaid-5-swivel-plate-caster', 'sku': None, 'name': None} 2024-09-18 23:27:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:29 2024-09-18 23:27:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:30 2024-09-18 23:27:46 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:31 2024-09-18 23:27:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:32 2024-09-18 23:27:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:33 2024-09-18 23:27:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:34 2024-09-18 23:27:50 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:35 2024-09-18 23:27:51 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:36 2024-09-18 23:27:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:37 2024-09-18 23:27:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:38 2024-09-18 23:27:55 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:39 2024-09-18 23:27:56 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:40 2024-09-18 23:27:56 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/boost-xl-eva-protection-case-lightweight-durable-weather-resistant-gbc017', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/boost-xl-eva-protection-case-lightweight-durable-weather-resistant-gbc017', 'sku': None, 'name': None} 2024-09-18 23:27:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:41 2024-09-18 23:27:58 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:42 2024-09-18 23:28:00 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:43 2024-09-18 23:28:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:44 2024-09-18 23:28:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:45 2024-09-18 23:28:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:46 2024-09-18 23:28:04 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:47 2024-09-18 23:28:04 [scrapy.extensions.logstats] INFO: Crawled 8 pages (at 8 pages/min), scraped 118698 items (at 118698 items/min) 2024-09-18 23:28:05 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:48 2024-09-18 23:28:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:49 2024-09-18 23:28:08 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:50 2024-09-18 23:28:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:51 2024-09-18 23:28:10 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:52 2024-09-18 23:28:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:53 2024-09-18 23:28:13 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:54 2024-09-18 23:28:13 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/heavy-duty-pro-universal-air-operated-oil-transfer-pump', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/heavy-duty-pro-universal-air-operated-oil-transfer-pump', 'sku': None, 'name': None} 2024-09-18 23:28:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:55 2024-09-18 23:28:14 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/led-permanent-mount-or-1-npt-pipe-mount-warning-light-amber-lens-3080ppm-a', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/led-permanent-mount-or-1-npt-pipe-mount-warning-light-amber-lens-3080ppm-a', 'sku': None, 'name': None} 2024-09-18 23:28:14 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/led-permanent-mount-warning-light-quad-flash-12-100-volt-green-lens-3077p-g', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/led-permanent-mount-warning-light-quad-flash-12-100-volt-green-lens-3077p-g', 'sku': None, 'name': None} 2024-09-18 23:28:14 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/champion-fmpsa5411mc-fmpro-all-stainless-steel-54-wx20-1-4-dx43-1-8-h-11-drawer-roller-cabinet', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/champion-fmpsa5411mc-fmpro-all-stainless-steel-54-wx20-1-4-dx43-1-8-h-11-drawer-roller-cabinet', 'sku': None, 'name': None} 2024-09-18 23:28:15 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:56 2024-09-18 23:28:16 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:57 2024-09-18 23:28:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:58 2024-09-18 23:28:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:59 2024-09-18 23:28:19 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cutting-edge-96-x-60-x-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cutting-edge-96-x-60-x-500', 'sku': None, 'name': None} 2024-09-18 23:28:19 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cutting-edge-108-x-60-x-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cutting-edge-108-x-60-x-500', 'sku': None, 'name': None} 2024-09-18 23:28:19 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cutting-edge-90-x-60-x-375', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cutting-edge-90-x-60-x-375', 'sku': None, 'name': None} 2024-09-18 23:28:19 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cutting-edge-90-x-60-x-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cutting-edge-90-x-60-x-500', 'sku': None, 'name': None} 2024-09-18 23:28:19 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cutting-edge-96-x-60-x-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cutting-edge-96-x-60-x-500', 'sku': None, 'name': None} 2024-09-18 23:28:19 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cutting-edge-108-x-60-x-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cutting-edge-108-x-60-x-500', 'sku': None, 'name': None} 2024-09-18 23:28:19 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cutting-edge-90-x-60-x-375', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cutting-edge-90-x-60-x-375', 'sku': None, 'name': None} 2024-09-18 23:28:19 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cutting-edge-102-x-60-x-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cutting-edge-102-x-60-x-500', 'sku': None, 'name': None} 2024-09-18 23:28:19 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cutting-edge-90-x-60-x-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cutting-edge-90-x-60-x-500', 'sku': None, 'name': None} 2024-09-18 23:28:19 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cutting-edge-96-x-60-x-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cutting-edge-96-x-60-x-500', 'sku': None, 'name': None} 2024-09-18 23:28:19 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cutting-edge-90-x-60-x-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cutting-edge-90-x-60-x-500', 'sku': None, 'name': None} 2024-09-18 23:28:19 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cutting-edge-96-x-60-x-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cutting-edge-96-x-60-x-500', 'sku': None, 'name': None} 2024-09-18 23:28:19 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cutting-edge-102-x-60-x-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cutting-edge-102-x-60-x-500', 'sku': None, 'name': None} 2024-09-18 23:28:19 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cutting-edge-108-x-60-x-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cutting-edge-108-x-60-x-500', 'sku': None, 'name': None} 2024-09-18 23:28:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:60 2024-09-18 23:28:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:61 2024-09-18 23:28:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:62 2024-09-18 23:28:25 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/dynabrade-3-80-grit-dynacut-extreme-orange-film-sanding-disc-50-count-box', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/dynabrade-3-80-grit-dynacut-extreme-orange-film-sanding-disc-50-count-box', 'sku': None, 'name': None} 2024-09-18 23:28:25 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/dynabrade-3-100-grit-dynacut-extreme-orange-film-sanding-disc-50-count-box', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/dynabrade-3-100-grit-dynacut-extreme-orange-film-sanding-disc-50-count-box', 'sku': None, 'name': None} 2024-09-18 23:28:25 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/dynabrade-3-120-grit-dynacut-extreme-orange-film-sanding-disc-50-count-box', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/dynabrade-3-120-grit-dynacut-extreme-orange-film-sanding-disc-50-count-box', 'sku': None, 'name': None} 2024-09-18 23:28:25 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/dynabrade-3-150-grit-dynacut-extreme-orange-film-sanding-disc-50-count-box', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/dynabrade-3-150-grit-dynacut-extreme-orange-film-sanding-disc-50-count-box', 'sku': None, 'name': None} 2024-09-18 23:28:25 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/dynabrade-3-180-grit-dynacut-extreme-orange-film-sanding-disc-50-count-box', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/dynabrade-3-180-grit-dynacut-extreme-orange-film-sanding-disc-50-count-box', 'sku': None, 'name': None} 2024-09-18 23:28:25 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/dynabrade-3-220-grit-dynacut-extreme-orange-film-sanding-disc-50-count-box', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/dynabrade-3-220-grit-dynacut-extreme-orange-film-sanding-disc-50-count-box', 'sku': None, 'name': None} 2024-09-18 23:28:25 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/dynabrade-3-320-grit-dynacut-extreme-orange-film-sanding-disc-50-count-box', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/dynabrade-3-320-grit-dynacut-extreme-orange-film-sanding-disc-50-count-box', 'sku': None, 'name': None} 2024-09-18 23:28:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:63 2024-09-18 23:28:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:64 2024-09-18 23:28:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:65 2024-09-18 23:28:29 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:66 2024-09-18 23:28:31 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:67 2024-09-18 23:28:31 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/1-inch-drive-size-impact-wrench-2000-ft-lb-maxtorque', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/1-inch-drive-size-impact-wrench-2000-ft-lb-maxtorque', 'sku': None, 'name': None} 2024-09-18 23:28:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:68 2024-09-18 23:28:34 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:69 2024-09-18 23:28:38 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/bulkpak-folding-bulk-shipping-container-hd-lid-48-inch-l-x-40-inch-w-black', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/bulkpak-folding-bulk-shipping-container-hd-lid-48-inch-l-x-40-inch-w-black', 'sku': None, 'name': None} 2024-09-18 23:28:38 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:70 2024-09-18 23:28:38 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/ultra-strong-scented-tall-white-kitchen-bags-13-gal-0-9-mil-white-80-box', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/ultra-strong-scented-tall-white-kitchen-bags-13-gal-0-9-mil-white-80-box', 'sku': None, 'name': None} 2024-09-18 23:28:38 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/ultra-strong-scented-tall-white-kitchen-bags-13-gal-0-9-mil-white-240-carton', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/ultra-strong-scented-tall-white-kitchen-bags-13-gal-0-9-mil-white-240-carton', 'sku': None, 'name': None} 2024-09-18 23:28:38 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/red-dot-ecg-monitoring-electrodes-micropore-tape-backing-2-4-inch-dia-50-bag-20-bag-case', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/red-dot-ecg-monitoring-electrodes-micropore-tape-backing-2-4-inch-dia-50-bag-20-bag-case', 'sku': None, 'name': None} 2024-09-18 23:28:40 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:71 2024-09-18 23:28:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:72 2024-09-18 23:28:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:73 2024-09-18 23:28:44 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/ez-gear-sensor-retro-fit', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/ez-gear-sensor-retro-fit', 'sku': None, 'name': None} 2024-09-18 23:28:44 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/relief-valve-closet', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/relief-valve-closet', 'sku': None, 'name': None} 2024-09-18 23:28:45 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:74 2024-09-18 23:28:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:75 2024-09-18 23:28:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:76 2024-09-18 23:28:50 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:77 2024-09-18 23:28:52 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/arsenal-bucket-truck-tool-bag-w-bucket-hooks-kit-gray', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/arsenal-bucket-truck-tool-bag-w-bucket-hooks-kit-gray', 'sku': None, 'name': None} 2024-09-18 23:28:52 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/3178-locking-bucket-hook-w-tethering-point-single-2-40-lb-cap-black-orange', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/3178-locking-bucket-hook-w-tethering-point-single-2-40-lb-cap-black-orange', 'sku': None, 'name': None} 2024-09-18 23:28:52 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/3178-locking-bucket-hook-w-tethering-point-single-3-40-lb-cap-black-orange', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/3178-locking-bucket-hook-w-tethering-point-single-3-40-lb-cap-black-orange', 'sku': None, 'name': None} 2024-09-18 23:28:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:78 2024-09-18 23:28:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:79 2024-09-18 23:28:55 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/rear-connect-diverter-tub-spout-12-x-34-rear-lift', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/rear-connect-diverter-tub-spout-12-x-34-rear-lift', 'sku': None, 'name': None} 2024-09-18 23:28:55 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/brass-shower-head-25-gpm', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/brass-shower-head-25-gpm', 'sku': None, 'name': None} 2024-09-18 23:28:55 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/lavatory-faucet-w-pop-up-12-gpm-satin-nickel', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/lavatory-faucet-w-pop-up-12-gpm-satin-nickel', 'sku': None, 'name': None} 2024-09-18 23:28:55 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:80 2024-09-18 23:28:56 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/fortress-755fs-mobility-scooter-replacement-battery-35ah-12v-nb', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/fortress-755fs-mobility-scooter-replacement-battery-35ah-12v-nb', 'sku': None, 'name': None} 2024-09-18 23:28:57 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/disinfecting-wipes-fresh-scent-citrus-blend-75-wipescanister-pack-of-3', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/disinfecting-wipes-fresh-scent-citrus-blend-75-wipescanister-pack-of-3', 'sku': None, 'name': None} 2024-09-18 23:28:57 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/healthcare-cleaner-disinfectant-wipes-unscented-185-wipespack-2-packcarton', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/healthcare-cleaner-disinfectant-wipes-unscented-185-wipespack-2-packcarton', 'sku': None, 'name': None} 2024-09-18 23:28:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:81 2024-09-18 23:28:58 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/sterile-sampling-bag-4-oz-capacity-7l-x-3w-pack-of-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/sterile-sampling-bag-4-oz-capacity-7l-x-3w-pack-of-500', 'sku': None, 'name': None} 2024-09-18 23:28:59 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/sterile-sampling-bag-18-oz-capacity-9l-x-4-12w-pack-of-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/sterile-sampling-bag-18-oz-capacity-9l-x-4-12w-pack-of-500', 'sku': None, 'name': None} 2024-09-18 23:28:59 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/sterile-sampling-bag-55-oz-capacity-11-1316l-x-7-116w-pack-of-500', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/sterile-sampling-bag-55-oz-capacity-11-1316l-x-7-116w-pack-of-500', 'sku': None, 'name': None} 2024-09-18 23:28:59 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:82 2024-09-18 23:29:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:83 2024-09-18 23:29:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:84 2024-09-18 23:29:03 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/1-air-impact-wrench-2000-ft-lbs-torque-pistol-grip', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/1-air-impact-wrench-2000-ft-lbs-torque-pistol-grip', 'sku': None, 'name': None} 2024-09-18 23:29:03 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/digger-78-x-3-14-hex-shank-2350-bpm-25-stroke-length-169-bore', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/digger-78-x-3-14-hex-shank-2350-bpm-25-stroke-length-169-bore', 'sku': None, 'name': None} 2024-09-18 23:29:03 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/digger-78-x-3-14-hex-shank-1850-bpm-4-stroke-length-169-bore', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/digger-78-x-3-14-hex-shank-1850-bpm-4-stroke-length-169-bore', 'sku': None, 'name': None} 2024-09-18 23:29:03 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/digger-78-x-3-14-hex-shank-2350-bpm-25-stroke-length-169-bore', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/digger-78-x-3-14-hex-shank-2350-bpm-25-stroke-length-169-bore', 'sku': None, 'name': None} 2024-09-18 23:29:03 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/pavement-breaker-1-14-x-6-hex-shank-575-stroke-length-262-bore', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/pavement-breaker-1-14-x-6-hex-shank-575-stroke-length-262-bore', 'sku': None, 'name': None} 2024-09-18 23:29:04 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/pavement-breaker-1-18-x-6-hex-shank-575-stroke-length-262-bore', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/pavement-breaker-1-18-x-6-hex-shank-575-stroke-length-262-bore', 'sku': None, 'name': None} 2024-09-18 23:29:04 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/pavement-breaker-1-18-x-6-hex-shank-625-stroke-length-1-34-bore', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/pavement-breaker-1-18-x-6-hex-shank-625-stroke-length-1-34-bore', 'sku': None, 'name': None} 2024-09-18 23:29:04 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/pavement-breaker-1-x-4-14-hex-shank-625-stroke-length-1-34-bore', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/pavement-breaker-1-x-4-14-hex-shank-625-stroke-length-1-34-bore', 'sku': None, 'name': None} 2024-09-18 23:29:04 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/chipping-hammer-68-round-shank-4-stroke-1480-bpm', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/chipping-hammer-68-round-shank-4-stroke-1480-bpm', 'sku': None, 'name': None} 2024-09-18 23:29:04 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/chipping-hammer-58-hex-shank-2-stroke-2300-bpm', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/chipping-hammer-58-hex-shank-2-stroke-2300-bpm', 'sku': None, 'name': None} 2024-09-18 23:29:04 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/chipping-hammer-58-hex-shank-3-stroke-1725-bpm', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/chipping-hammer-58-hex-shank-3-stroke-1725-bpm', 'sku': None, 'name': None} 2024-09-18 23:29:04 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/chipping-hammer-58-hex-shank-4-stroke-1480-bpm', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/chipping-hammer-58-hex-shank-4-stroke-1480-bpm', 'sku': None, 'name': None} 2024-09-18 23:29:04 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/air-care-impact-wrench-lubrication-kit', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/air-care-impact-wrench-lubrication-kit', 'sku': None, 'name': None} 2024-09-18 23:29:04 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:85 2024-09-18 23:29:04 [scrapy.extensions.logstats] INFO: Crawled 13 pages (at 5 pages/min), scraped 213627 items (at 94929 items/min) 2024-09-18 23:29:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:86 2024-09-18 23:29:06 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/2-1-2-spring-action-belt-clip-for-xpr-series-radios', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/2-1-2-spring-action-belt-clip-for-xpr-series-radios', 'sku': None, 'name': None} 2024-09-18 23:29:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:87 2024-09-18 23:29:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:88 2024-09-18 23:29:10 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/safety-zlink-kit-fr-face-seal-c40', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/safety-zlink-kit-fr-face-seal-c40', 'sku': None, 'name': None} 2024-09-18 23:29:10 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/safety-zlink-fr-shoulder-cape-px5-papr', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/safety-zlink-fr-shoulder-cape-px5-papr', 'sku': None, 'name': None} 2024-09-18 23:29:10 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/safety-zlink-fr-face-seal-px5-papr', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/safety-zlink-fr-face-seal-px5-papr', 'sku': None, 'name': None} 2024-09-18 23:29:11 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/safety-z4-fr-face-seal', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/safety-z4-fr-face-seal', 'sku': None, 'name': None} 2024-09-18 23:29:11 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/safety-z4-fr-shoulder-cape', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/safety-z4-fr-shoulder-cape', 'sku': None, 'name': None} 2024-09-18 23:29:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:89 2024-09-18 23:29:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:90 2024-09-18 23:29:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:91 2024-09-18 23:29:15 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/sj3571-loop-fastener-1w-x-150l-white', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/sj3571-loop-fastener-1w-x-150l-white', 'sku': None, 'name': None} 2024-09-18 23:29:15 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/sj3572-hook-fastener-1w-x-150l-white', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/sj3572-hook-fastener-1w-x-150l-white', 'sku': None, 'name': None} 2024-09-18 23:29:15 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/mp3560-mini-pack-fasteners-1w-x-15l-clear', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/mp3560-mini-pack-fasteners-1w-x-15l-clear', 'sku': None, 'name': None} 2024-09-18 23:29:15 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/sj4570-low-profile-fasteners-1w-x-150l-clear', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/sj4570-low-profile-fasteners-1w-x-150l-clear', 'sku': None, 'name': None} 2024-09-18 23:29:15 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/mp3526nmp3527n-hook--loop-combo-pack-fasteners-1w-x-15l-black', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/mp3526nmp3527n-hook--loop-combo-pack-fasteners-1w-x-15l-black', 'sku': None, 'name': None} 2024-09-18 23:29:15 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/sj4570-low-profile-fasteners-58w-x-150l-clear', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/sj4570-low-profile-fasteners-58w-x-150l-clear', 'sku': None, 'name': None} 2024-09-18 23:29:15 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/sj3540-fasteners-1w-x-150l-black', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/sj3540-fasteners-1w-x-150l-black', 'sku': None, 'name': None} 2024-09-18 23:29:15 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/sj3541-fasteners-1w-x-150l-black', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/sj3541-fasteners-1w-x-150l-black', 'sku': None, 'name': None} 2024-09-18 23:29:15 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/sj3560-fasteners-1w-x-150l-clear', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/sj3560-fasteners-1w-x-150l-clear', 'sku': None, 'name': None} 2024-09-18 23:29:15 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/sj3571-loop-fastener-1w-x-150l-black', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/sj3571-loop-fastener-1w-x-150l-black', 'sku': None, 'name': None} 2024-09-18 23:29:16 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:92 2024-09-18 23:29:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:93 2024-09-18 23:29:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:94 2024-09-18 23:29:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:95 2024-09-18 23:29:21 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/as-20x-50x-kid-s-deluxe-stereo-microscope-with-digital-camera-fossil-kit-dual-illumination', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/as-20x-50x-kid-s-deluxe-stereo-microscope-with-digital-camera-fossil-kit-dual-illumination', 'sku': None, 'name': None} 2024-09-18 23:29:21 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/as-40x-1000x-monocular-microscope-25-pc-slides-set-tweezers-plus-usb-digital-microscope', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/as-40x-1000x-monocular-microscope-25-pc-slides-set-tweezers-plus-usb-digital-microscope', 'sku': None, 'name': None} 2024-09-18 23:29:21 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/as-20x-50x-portable-stereo-microscope-6-pc-3d-insect-specimen-kits-book-digital-camera', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/as-20x-50x-portable-stereo-microscope-6-pc-3d-insect-specimen-kits-book-digital-camera', 'sku': None, 'name': None} 2024-09-18 23:29:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:96 2024-09-18 23:29:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:97 2024-09-18 23:29:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:98 2024-09-18 23:29:25 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/manway-gasket-spiral-wound-11-x-15-x-1516-0175-thick-elliptical-pk-3', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/manway-gasket-spiral-wound-11-x-15-x-1516-0175-thick-elliptical-pk-3', 'sku': None, 'name': None} 2024-09-18 23:29:25 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/manway-gasket-spiral-wound-12-x-16-x-1516-025-thick-elliptical-pk-3', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/manway-gasket-spiral-wound-12-x-16-x-1516-025-thick-elliptical-pk-3', 'sku': None, 'name': None} 2024-09-18 23:29:25 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/manway-gasket-spiral-wound-11-x-15-x-114-0175-thick-elliptical-pk-3', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/manway-gasket-spiral-wound-11-x-15-x-114-0175-thick-elliptical-pk-3', 'sku': None, 'name': None} 2024-09-18 23:29:25 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/manway-gasket-spiral-wound-11-x-15-x-1516-025-thick-elliptical-pk-3', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/manway-gasket-spiral-wound-11-x-15-x-1516-025-thick-elliptical-pk-3', 'sku': None, 'name': None} 2024-09-18 23:29:25 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/manway-gasket-spiral-wound-11-x-15-x-114-025-thick-elliptical-pk-3', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/manway-gasket-spiral-wound-11-x-15-x-114-025-thick-elliptical-pk-3', 'sku': None, 'name': None} 2024-09-18 23:29:25 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/manway-gasket-spiral-wound-12-x-16-x-1516-0175-thick-elliptical-pk-3', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/manway-gasket-spiral-wound-12-x-16-x-1516-0175-thick-elliptical-pk-3', 'sku': None, 'name': None} 2024-09-18 23:29:25 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/manway-gasket-spiral-wound-12-x-16-x-114-0175-thick-elliptical-pk-3', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/manway-gasket-spiral-wound-12-x-16-x-114-0175-thick-elliptical-pk-3', 'sku': None, 'name': None} 2024-09-18 23:29:26 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/8020-15-series-3-x-34-bi-t-slotted-extrusion-profile-120l-stock-bar', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/8020-15-series-3-x-34-bi-t-slotted-extrusion-profile-120l-stock-bar', 'sku': None, 'name': None} 2024-09-18 23:29:26 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/topoge-series-180-handhole-gasket-314-x-412-x-916-black-rubber-obround-2-pack', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/topoge-series-180-handhole-gasket-314-x-412-x-916-black-rubber-obround-2-pack', 'sku': None, 'name': None} 2024-09-18 23:29:26 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/topoge-series-180-handhole-gasket-314-x-412-x-916-black-rubber-obround-2-pack', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/topoge-series-180-handhole-gasket-314-x-412-x-916-black-rubber-obround-2-pack', 'sku': None, 'name': None} 2024-09-18 23:29:27 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/ediusa-ballistic-tshirt-tested-to-level-iiia-ballistic-resistance-xxlarge', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/ediusa-ballistic-tshirt-tested-to-level-iiia-ballistic-resistance-xxlarge', 'sku': None, 'name': None} 2024-09-18 23:29:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:99 2024-09-18 23:29:29 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:100 2024-09-18 23:29:30 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/bi-metallic-surface-temperature-thermometer-20-260c-0-500f-50mm-2-dial', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/bi-metallic-surface-temperature-thermometer-20-260c-0-500f-50mm-2-dial', 'sku': None, 'name': None} 2024-09-18 23:29:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:101 2024-09-18 23:29:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:102 2024-09-18 23:29:34 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:103 2024-09-18 23:29:34 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/drywall-screw-phillips-bugle-head-fine-thread-6-x-1-14-size-pack-of-100', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/drywall-screw-phillips-bugle-head-fine-thread-6-x-1-14-size-pack-of-100', 'sku': None, 'name': None} 2024-09-18 23:29:34 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/drywall-screw-phillips-bugle-head-fine-thread-6-x-1-14-size-tuff-pack-of-100', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/drywall-screw-phillips-bugle-head-fine-thread-6-x-1-14-size-tuff-pack-of-100', 'sku': None, 'name': None} 2024-09-18 23:29:34 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/drywall-screw-phillips-bugle-head-fine-thread-6-x-1-58-size-tuff-pack-of-100', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/drywall-screw-phillips-bugle-head-fine-thread-6-x-1-58-size-tuff-pack-of-100', 'sku': None, 'name': None} 2024-09-18 23:29:34 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/drywall-screw-phillips-bugle-head-fine-thread-6-x-2-size-tuff-pack-of-100', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/drywall-screw-phillips-bugle-head-fine-thread-6-x-2-size-tuff-pack-of-100', 'sku': None, 'name': None} 2024-09-18 23:29:35 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/drywall-screw-phillips-bugle-head-fine-thread-6-x-2-size-pack-of-100', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/drywall-screw-phillips-bugle-head-fine-thread-6-x-2-size-pack-of-100', 'sku': None, 'name': None} 2024-09-18 23:29:35 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/drywall-screw-phillips-bugle-head-fine-thread-6-x-1-58-size-pack-of-100', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/drywall-screw-phillips-bugle-head-fine-thread-6-x-1-58-size-pack-of-100', 'sku': None, 'name': None} 2024-09-18 23:29:35 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:104 2024-09-18 23:29:36 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/dbi-sala-nano-lok-xl-self-retracting-lifeline-dyneema-webbing--swivel-snap-hook-20l', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/dbi-sala-nano-lok-xl-self-retracting-lifeline-dyneema-webbing--swivel-snap-hook-20l', 'sku': None, 'name': None} 2024-09-18 23:29:36 [scrapy.core.downloader.handlers.http11] WARNING: Got data loss in https://www.globalindustrial.com/sitemap-products-3.xml. If you want to process broken responses set the setting DOWNLOAD_FAIL_ON_DATALOSS = False -- This message won't be shown in further requests 2024-09-18 23:29:37 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/quantum-partition-store-basket-cart-with-add-on-unit--wire-baskets-19-12l-x-24w-x-75h-chrome', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/quantum-partition-store-basket-cart-with-add-on-unit--wire-baskets-19-12l-x-24w-x-75h-chrome', 'sku': None, 'name': None} 2024-09-18 23:29:37 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:105 2024-09-18 23:29:38 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/ac-inpass-2-1-2-inch-npt-coalescing-filter-716-cfm-8-000-hr-0-1-micron-232-psi', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/ac-inpass-2-1-2-inch-npt-coalescing-filter-716-cfm-8-000-hr-0-1-micron-232-psi', 'sku': None, 'name': None} 2024-09-18 23:29:38 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/ac-inpass-2-inch-npt-coalescing-filter-573-cfm-8-000-hr-0-1-micron-232-psi', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/ac-inpass-2-inch-npt-coalescing-filter-573-cfm-8-000-hr-0-1-micron-232-psi', 'sku': None, 'name': None} 2024-09-18 23:29:38 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/ac-inpass-2-inch-npt-coalescing-filter-716-cfm-8-000-hr-0-1-micron-232-psi', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/ac-inpass-2-inch-npt-coalescing-filter-716-cfm-8-000-hr-0-1-micron-232-psi', 'sku': None, 'name': None} 2024-09-18 23:29:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:106 2024-09-18 23:29:40 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/42-inch-tow-lawn-sweeper-67-inch-l-x-51-inch-w-x-24-inch-h-black', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/42-inch-tow-lawn-sweeper-67-inch-l-x-51-inch-w-x-24-inch-h-black', 'sku': None, 'name': None} 2024-09-18 23:29:40 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/pvc-flagger-pole-for-stopslow-signs-spring-button-84-post-height-orange', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/pvc-flagger-pole-for-stopslow-signs-spring-button-84-post-height-orange', 'sku': None, 'name': None} 2024-09-18 23:29:41 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:107 2024-09-18 23:29:41 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/surgical-tape-2-x-10-yards-6-roll-box-12-box-case', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/surgical-tape-2-x-10-yards-6-roll-box-12-box-case', 'sku': None, 'name': None} 2024-09-18 23:29:41 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/surgical-tape-3-x-10-yards-4-roll-box-12-box-case', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/surgical-tape-3-x-10-yards-4-roll-box-12-box-case', 'sku': None, 'name': None} 2024-09-18 23:29:41 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/surgical-tape-1-2-x-10-yards-24-roll-box-12-box-case', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/surgical-tape-1-2-x-10-yards-24-roll-box-12-box-case', 'sku': None, 'name': None} 2024-09-18 23:29:41 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/cx1-series-provector-explosion-proof-convection-heater-with-tstat-3600w-480v-1-phase', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/cx1-series-provector-explosion-proof-convection-heater-with-tstat-3600w-480v-1-phase', 'sku': None, 'name': None} 2024-09-18 23:29:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:108 2024-09-18 23:29:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:109 2024-09-18 23:29:46 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:110 2024-09-18 23:29:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:111 2024-09-18 23:29:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:112 2024-09-18 23:29:51 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:113 2024-09-18 23:29:52 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/low-voltage-thermostat-non-programmable-1h-1c-single-stage', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/low-voltage-thermostat-non-programmable-1h-1c-single-stage', 'sku': None, 'name': None} 2024-09-18 23:29:52 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/low-voltage-thermostat-programmable-7-day-or-5-1-1-1h-1c-single-stage', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/low-voltage-thermostat-programmable-7-day-or-5-1-1-1h-1c-single-stage', 'sku': None, 'name': None} 2024-09-18 23:29:52 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/low-voltage-thermostat-non-programmable-1h-1c-single-stage', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/low-voltage-thermostat-non-programmable-1h-1c-single-stage', 'sku': None, 'name': None} 2024-09-18 23:29:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:114 2024-09-18 23:29:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:115 2024-09-18 23:30:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:116 2024-09-18 23:30:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:117 2024-09-18 23:30:04 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:118 2024-09-18 23:30:04 [scrapy.extensions.logstats] INFO: Crawled 14 pages (at 1 pages/min), scraped 296357 items (at 82730 items/min) 2024-09-18 23:30:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:119 2024-09-18 23:30:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:120 2024-09-18 23:30:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:121 2024-09-18 23:30:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:122 2024-09-18 23:30:11 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/traditional-single-tier-locker-1-wide-12w-x-12d-x-66h-assembled-black', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/traditional-single-tier-locker-1-wide-12w-x-12d-x-66h-assembled-black', 'sku': None, 'name': None} 2024-09-18 23:30:11 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/traditional-single-tier-locker-1-wide-12w-x-12d-x-66h-assembled-black', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/traditional-single-tier-locker-1-wide-12w-x-12d-x-66h-assembled-black', 'sku': None, 'name': None} 2024-09-18 23:30:11 [scrapy.core.scraper] WARNING: Dropped: Duplicate item found: {'url': 'https://www.globalindustrial.com/p/traditional-single-tier-locker-1-wide-12w-x-12d-x-66h-assembled-black', 'sku': None, 'name': None} {'url': 'https://www.globalindustrial.com/p/traditional-single-tier-locker-1-wide-12w-x-12d-x-66h-assembled-black', 'sku': None, 'name': None} 2024-09-18 23:30:13 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:123 2024-09-18 23:30:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:124 2024-09-18 23:30:16 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:125 2024-09-18 23:30:16 [scrapy.core.engine] INFO: Closing spider (finished) 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] ------------------------------ MONITORS ------------------------------ 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] Extracted Items Monitor/test_stat_monitor... OK 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] Item Validation Monitor/test_stat_monitor... SKIPPED (Unable to find 'spidermon/validation/fields/errors' in job stats.) 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] Error Count Monitor/test_stat_monitor... SKIPPED (Unable to find 'log_count/ERROR' in job stats.) 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] Warning Count Monitor/test_stat_monitor... FAIL 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] Finish Reason Monitor/Should have the expected finished reason(s)... OK 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] Unwanted HTTP codes monitor/Should not hit the limit of unwanted http status... OK 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] Field Coverage Monitor/test_check_if_field_coverage_rules_are_met... OK 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] Retry Count monitor/Should not hit the limit of requests that reached the maximum retry amount... OK 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] Downloader Exceptions monitor/test_stat_monitor... OK 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] Successful Requests monitor/Should have at least the minimum number of successful requests... OK 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] Total Requests monitor/Should not hit the total limit of requests... OK 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] ---------------------------------------------------------------------- 2024-09-18 23:30:16 [global_industrial] ERROR: [Spidermon] ====================================================================== FAIL: Warning Count Monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 184, in test_stat_monitor assertion_method( AssertionError: Expecting 'log_count/WARNING' to be '<=' to '1.0'. Current value: '130' 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] 11 monitors in 0.002s 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] FAILED (failures=1, skipped=2) 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] -------------------------- FINISHED ACTIONS -------------------------- 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] ---------------------------------------------------------------------- 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] 0 actions in 0.000s 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] OK 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] --------------------------- PASSED ACTIONS --------------------------- 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] ---------------------------------------------------------------------- 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] 0 actions in 0.000s 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] OK 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] --------------------------- FAILED ACTIONS --------------------------- 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] ---------------------------------------------------------------------- 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] 0 actions in 0.000s 2024-09-18 23:30:16 [global_industrial] INFO: [Spidermon] OK 2024-09-18 23:30:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (1294 items) in: bq://response-elt.dev_scrapers.catalog_urls/batch:126 2024-09-18 23:30:17 [scrapy.statscollectors] INFO: Dumping Scrapy stats: {'DuplicatesFilterPipeline/duplicates_filtered': 123, 'downloader/exception_count': 7, 'downloader/exception_type_count/twisted.web._newclient.ResponseFailed': 7, 'downloader/request_bytes': 14303, 'downloader/request_count': 21, 'downloader/request_method_count/GET': 21, 'downloader/response_bytes': 3596621, 'downloader/response_count': 14, 'downloader/response_status_count/200': 14, 'elapsed_time_seconds': 192.09416, 'feedexport/success_count/BigQueryFeedStorage': 126, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2024, 9, 18, 23, 30, 16, 957612, tzinfo=datetime.timezone.utc), 'httpcompression/response_bytes': 65381998, 'httpcompression/response_count': 14, 'item_dropped_count': 123, 'item_dropped_reasons_count/DropItem': 123, 'item_scraped_count': 313794, 'log_count/ERROR': 1, 'log_count/INFO': 167, 'log_count/WARNING': 130, 'memusage/max': 620924928, 'memusage/startup': 114155520, 'poet/injector/catalog_discovery.pages.global_industrial.GlobalIndustrialSitemapPageObject': 14, 'request_depth_max': 1, 'response_received_count': 14, 'retry/count': 7, 'retry/reason_count/twisted.web._newclient.ResponseFailed': 7, 'scheduler/dequeued': 21, 'scheduler/dequeued/memory': 21, 'scheduler/enqueued': 21, 'scheduler/enqueued/memory': 21, 'spidermon/validation/fields': 1255176, 'spidermon/validation/items': 313794, 'spidermon/validation/validators': 1, 'spidermon/validation/validators/item/jsonschema': True, 'spidermon_field_coverage/dict/name': 1.0, 'spidermon_field_coverage/dict/sku': 1.0, 'spidermon_field_coverage/dict/supplier': 1.0, 'spidermon_field_coverage/dict/url': 1.0, 'spidermon_item_scraped_count': 313794, 'spidermon_item_scraped_count/dict': 313794, 'spidermon_item_scraped_count/dict/name': 313794, 'spidermon_item_scraped_count/dict/sku': 313794, 'spidermon_item_scraped_count/dict/supplier': 313794, 'spidermon_item_scraped_count/dict/url': 313794, 'start_time': datetime.datetime(2024, 9, 18, 23, 27, 4, 863452, tzinfo=datetime.timezone.utc)} 2024-09-18 23:30:17 [scrapy.core.engine] INFO: Spider closed (finished)