2024-08-01 04:43:46 [scrapy.utils.log] INFO: Scrapy 2.11.2 started (bot: catalog_discovery)
2024-08-01 04:43:46 [scrapy.utils.log] INFO: Versions: lxml 5.2.2.0, libxml2 2.12.6, cssselect 1.2.0, parsel 1.9.1, w3lib 2.1.2, Twisted 24.3.0, Python 3.11.9 (main, Jul 23 2024, 07:22:56) [GCC 12.2.0], pyOpenSSL 24.1.0 (OpenSSL 3.2.1 30 Jan 2024), cryptography 42.0.7, Platform Linux-6.4.10-dirty-x86_64-with-glibc2.36
2024-08-01 04:43:46 [scrapy.addons] INFO: Enabled addons:
[]
2024-08-01 04:43:46 [scrapy.extensions.telnet] INFO: Telnet Password: 531c1f8ef872b637
2024-08-01 04:43:46 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats',
 'spidermon.contrib.scrapy.extensions.Spidermon']
2024-08-01 04:43:46 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'catalog_discovery',
 'CONCURRENT_ITEMS': 1000,
 'CONCURRENT_REQUESTS': 32,
 'FEED_EXPORT_ENCODING': 'utf-8',
 'LOG_FILE': '/var/lib/scrapyd/logs/catalog_discovery/grainger/a0ca2fa04fc011efae8d4200a9fe0102.log',
 'LOG_LEVEL': 'INFO',
 'NEWSPIDER_MODULE': 'catalog_discovery.spiders',
 'REQUEST_FINGERPRINTER_CLASS': 'scrapy_poet.ScrapyPoetRequestFingerprinter',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'RETRY_TIMES': 5,
 'SPIDER_MODULES': ['catalog_discovery.spiders'],
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor',
 'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:125.0) '
               'Gecko/20100101 Firefox/125.0'}
2024-08-01 04:43:47 [scrapy_poet.injection] INFO: Loading providers:
 [<class 'scrapy_poet.page_input_providers.HttpRequestProvider'>,
 <class 'scrapy_poet.page_input_providers.HttpResponseProvider'>,
 <class 'scrapy_poet.page_input_providers.HttpClientProvider'>,
 <class 'scrapy_poet.page_input_providers.PageParamsProvider'>,
 <class 'scrapy_poet.page_input_providers.RequestUrlProvider'>,
 <class 'scrapy_poet.page_input_providers.ResponseUrlProvider'>,
 <class 'scrapy_poet.page_input_providers.StatsProvider'>]
2024-08-01 04:43:47 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy_poet.InjectionMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy_poet.DownloaderStatsMiddleware']
2024-08-01 04:43:47 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy_poet.RetryMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2024-08-01 04:43:47 [scrapy.middleware] INFO: Enabled item pipelines:
['scraping_utils.pipelines.DuplicatesFilterPipeline',
 'scraping_utils.pipelines.AttachSupplierPipeline',
 'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline']
2024-08-01 04:43:47 [scrapy.core.engine] INFO: Spider opened
2024-08-01 04:43:47 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-08-01 04:43:47 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2024-08-01 04:43:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:1
2024-08-01 04:43:58 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:2
2024-08-01 04:43:59 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:3
2024-08-01 04:44:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:4
2024-08-01 04:44:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:5
2024-08-01 04:44:04 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:6
2024-08-01 04:44:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:7
2024-08-01 04:44:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:8
2024-08-01 04:44:08 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:9
2024-08-01 04:44:10 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:10
2024-08-01 04:44:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:11
2024-08-01 04:44:13 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:12
2024-08-01 04:44:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:13
2024-08-01 04:44:16 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:14
2024-08-01 04:44:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:15
2024-08-01 04:44:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:16
2024-08-01 04:44:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:17
2024-08-01 04:44:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:18
2024-08-01 04:44:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:19
2024-08-01 04:44:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:20
2024-08-01 04:44:26 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:21
2024-08-01 04:44:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:22
2024-08-01 04:44:29 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:23
2024-08-01 04:44:31 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:24
2024-08-01 04:44:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:25
2024-08-01 04:44:34 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:26
2024-08-01 04:44:35 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:27
2024-08-01 04:44:37 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:28
2024-08-01 04:44:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:29
2024-08-01 04:44:41 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:30
2024-08-01 04:44:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:31
2024-08-01 04:44:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:32
2024-08-01 04:44:45 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:33
2024-08-01 04:44:46 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:34
2024-08-01 04:44:47 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 29 pages/min), scraped 87919 items (at 87919 items/min)
2024-08-01 04:44:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:35
2024-08-01 04:44:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:36
2024-08-01 04:44:51 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:37
2024-08-01 04:44:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:38
2024-08-01 04:44:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:39
2024-08-01 04:44:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:40
2024-08-01 04:44:56 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:41
2024-08-01 04:44:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:42
2024-08-01 04:44:59 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:43
2024-08-01 04:45:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:44
2024-08-01 04:45:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:45
2024-08-01 04:45:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:46
2024-08-01 04:45:05 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:47
2024-08-01 04:45:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:48
2024-08-01 04:45:08 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:49
2024-08-01 04:45:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:50
2024-08-01 04:45:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:51
2024-08-01 04:45:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:52
2024-08-01 04:45:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:53
2024-08-01 04:45:15 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:54
2024-08-01 04:45:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:55
2024-08-01 04:45:18 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:56
2024-08-01 04:45:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:57
2024-08-01 04:45:21 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:58
2024-08-01 04:45:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:59
2024-08-01 04:45:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:60
2024-08-01 04:45:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:61
2024-08-01 04:45:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:62
2024-08-01 04:45:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:63
2024-08-01 04:45:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:64
2024-08-01 04:45:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:65
2024-08-01 04:45:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:66
2024-08-01 04:45:34 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:67
2024-08-01 04:45:36 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:68
2024-08-01 04:45:38 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:69
2024-08-01 04:45:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:70
2024-08-01 04:45:40 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:71
2024-08-01 04:45:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:72
2024-08-01 04:45:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:73
2024-08-01 04:45:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:74
2024-08-01 04:45:46 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:75
2024-08-01 04:45:47 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 189727 items (at 101808 items/min)
2024-08-01 04:45:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:76
2024-08-01 04:45:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:77
2024-08-01 04:45:50 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:78
2024-08-01 04:45:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:79
2024-08-01 04:45:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:80
2024-08-01 04:45:55 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:81
2024-08-01 04:45:56 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:82
2024-08-01 04:45:58 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:83
2024-08-01 04:45:59 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:84
2024-08-01 04:46:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:85
2024-08-01 04:46:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:86
2024-08-01 04:46:04 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:87
2024-08-01 04:46:05 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:88
2024-08-01 04:46:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:89
2024-08-01 04:46:08 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:90
2024-08-01 04:46:10 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:91
2024-08-01 04:46:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:92
2024-08-01 04:46:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:93
2024-08-01 04:46:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:94
2024-08-01 04:46:15 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:95
2024-08-01 04:46:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:96
2024-08-01 04:46:18 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:97
2024-08-01 04:46:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:98
2024-08-01 04:46:21 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:99
2024-08-01 04:46:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:100
2024-08-01 04:46:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:101
2024-08-01 04:46:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:102
2024-08-01 04:46:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:103
2024-08-01 04:46:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:104
2024-08-01 04:46:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:105
2024-08-01 04:46:31 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:106
2024-08-01 04:46:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:107
2024-08-01 04:46:35 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:108
2024-08-01 04:46:36 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:109
2024-08-01 04:46:38 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:110
2024-08-01 04:46:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:111
2024-08-01 04:46:41 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:112
2024-08-01 04:46:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:113
2024-08-01 04:46:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:114
2024-08-01 04:46:45 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:115
2024-08-01 04:46:46 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:116
2024-08-01 04:46:47 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 291367 items (at 101640 items/min)
2024-08-01 04:46:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:117
2024-08-01 04:46:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:118
2024-08-01 04:46:51 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:119
2024-08-01 04:46:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:120
2024-08-01 04:46:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:121
2024-08-01 04:46:55 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:122
2024-08-01 04:46:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:123
2024-08-01 04:46:58 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:124
2024-08-01 04:46:59 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:125
2024-08-01 04:47:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:126
2024-08-01 04:47:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:127
2024-08-01 04:47:04 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:128
2024-08-01 04:47:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:129
2024-08-01 04:47:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:130
2024-08-01 04:47:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:131
2024-08-01 04:47:10 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:132
2024-08-01 04:47:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:133
2024-08-01 04:47:13 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:134
2024-08-01 04:47:15 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:135
2024-08-01 04:47:16 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:136
2024-08-01 04:47:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:137
2024-08-01 04:47:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:138
2024-08-01 04:47:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:139
2024-08-01 04:47:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:140
2024-08-01 04:47:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:141
2024-08-01 04:47:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:142
2024-08-01 04:47:26 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:143
2024-08-01 04:47:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:144
2024-08-01 04:47:29 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:145
2024-08-01 04:47:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:146
2024-08-01 04:47:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:147
2024-08-01 04:47:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:148
2024-08-01 04:47:35 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:149
2024-08-01 04:47:36 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:150
2024-08-01 04:47:38 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:151
2024-08-01 04:47:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:152
2024-08-01 04:47:41 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:153
2024-08-01 04:47:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:154
2024-08-01 04:47:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:155
2024-08-01 04:47:45 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:156
2024-08-01 04:47:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:157
2024-08-01 04:47:47 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 393511 items (at 102144 items/min)
2024-08-01 04:47:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:158
2024-08-01 04:47:50 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:159
2024-08-01 04:47:51 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:160
2024-08-01 04:47:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:161
2024-08-01 04:47:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:162
2024-08-01 04:47:55 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:163
2024-08-01 04:47:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:164
2024-08-01 04:47:58 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:165
2024-08-01 04:48:00 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:166
2024-08-01 04:48:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:167
2024-08-01 04:48:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:168
2024-08-01 04:48:05 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:169
2024-08-01 04:48:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:170
2024-08-01 04:48:08 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:171
2024-08-01 04:48:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:172
2024-08-01 04:48:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:173
2024-08-01 04:48:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:174
2024-08-01 04:48:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:175
2024-08-01 04:48:15 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:176
2024-08-01 04:48:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:177
2024-08-01 04:48:18 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:178
2024-08-01 04:48:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:179
2024-08-01 04:48:21 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:180
2024-08-01 04:48:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:181
2024-08-01 04:48:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:182
2024-08-01 04:48:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:183
2024-08-01 04:48:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:184
2024-08-01 04:48:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:185
2024-08-01 04:48:29 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:186
2024-08-01 04:48:31 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:187
2024-08-01 04:48:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:188
2024-08-01 04:48:34 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:189
2024-08-01 04:48:36 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:190
2024-08-01 04:48:37 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:191
2024-08-01 04:48:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:192
2024-08-01 04:48:40 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:193
2024-08-01 04:48:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:194
2024-08-01 04:48:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:195
2024-08-01 04:48:45 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:196
2024-08-01 04:48:46 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:197
2024-08-01 04:48:47 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 494787 items (at 101276 items/min)
2024-08-01 04:48:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:198
2024-08-01 04:48:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:199
2024-08-01 04:48:50 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:200
2024-08-01 04:48:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:201
2024-08-01 04:48:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:202
2024-08-01 04:48:55 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:203
2024-08-01 04:48:56 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:204
2024-08-01 04:48:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:205
2024-08-01 04:48:59 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:206
2024-08-01 04:49:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:207
2024-08-01 04:49:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:208
2024-08-01 04:49:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:209
2024-08-01 04:49:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:210
2024-08-01 04:49:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:211
2024-08-01 04:49:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:212
2024-08-01 04:49:10 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:213
2024-08-01 04:49:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:214
2024-08-01 04:49:13 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:215
2024-08-01 04:49:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:216
2024-08-01 04:49:16 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:217
2024-08-01 04:49:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:218
2024-08-01 04:49:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:219
2024-08-01 04:49:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:220
2024-08-01 04:49:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:221
2024-08-01 04:49:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:222
2024-08-01 04:49:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:223
2024-08-01 04:49:26 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:224
2024-08-01 04:49:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:225
2024-08-01 04:49:29 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:226
2024-08-01 04:49:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:227
2024-08-01 04:49:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:228
2024-08-01 04:49:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:229
2024-08-01 04:49:36 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:230
2024-08-01 04:49:37 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:231
2024-08-01 04:49:38 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:232
2024-08-01 04:49:40 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:233
2024-08-01 04:49:41 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:234
2024-08-01 04:49:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:235
2024-08-01 04:49:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:236
2024-08-01 04:49:45 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:237
2024-08-01 04:49:47 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 595419 items (at 100632 items/min)
2024-08-01 04:49:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:238
2024-08-01 04:49:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:239
2024-08-01 04:49:51 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:240
2024-08-01 04:49:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:241
2024-08-01 04:49:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:242
2024-08-01 04:49:55 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:243
2024-08-01 04:49:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:244
2024-08-01 04:49:58 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:245
2024-08-01 04:49:59 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:246
2024-08-01 04:50:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:247
2024-08-01 04:50:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:248
2024-08-01 04:50:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:249
2024-08-01 04:50:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:250
2024-08-01 04:50:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:251
2024-08-01 04:50:08 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:252
2024-08-01 04:50:10 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:253
2024-08-01 04:50:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:254
2024-08-01 04:50:13 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:255
2024-08-01 04:50:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:256
2024-08-01 04:50:16 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:257
2024-08-01 04:50:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:258
2024-08-01 04:50:18 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:259
2024-08-01 04:50:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:260
2024-08-01 04:50:21 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:261
2024-08-01 04:50:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:262
2024-08-01 04:50:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:263
2024-08-01 04:50:26 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:264
2024-08-01 04:50:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:265
2024-08-01 04:50:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:266
2024-08-01 04:50:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:267
2024-08-01 04:50:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:268
2024-08-01 04:50:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:269
2024-08-01 04:50:35 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:270
2024-08-01 04:50:37 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:271
2024-08-01 04:50:38 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:272
2024-08-01 04:50:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:273
2024-08-01 04:50:41 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:274
2024-08-01 04:50:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:275
2024-08-01 04:50:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:276
2024-08-01 04:50:45 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:277
2024-08-01 04:50:47 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 696135 items (at 100716 items/min)
2024-08-01 04:50:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:278
2024-08-01 04:50:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:279
2024-08-01 04:50:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:280
2024-08-01 04:50:51 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:281
2024-08-01 04:50:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:282
2024-08-01 04:50:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:283
2024-08-01 04:50:56 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:284
2024-08-01 04:50:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:285
2024-08-01 04:50:58 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:286
2024-08-01 04:51:00 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:287
2024-08-01 04:51:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:288
2024-08-01 04:51:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:289
2024-08-01 04:51:05 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:290
2024-08-01 04:51:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:291
2024-08-01 04:51:08 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:292
2024-08-01 04:51:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:293
2024-08-01 04:51:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:294
2024-08-01 04:51:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:295
2024-08-01 04:51:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:296
2024-08-01 04:51:15 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:297
2024-08-01 04:51:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:298
2024-08-01 04:51:18 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:299
2024-08-01 04:51:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:300
2024-08-01 04:51:21 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:301
2024-08-01 04:51:21 [scrapy.core.scraper] WARNING: Dropped: Validation failed!
{'url': 'https://www.grainger.com/product/FLIR-Videoscope-1024-×-600-797RC4', 'sku': '797RC4', 'name': None, 'supplier': 'GRAINGER'}
2024-08-01 04:51:21 [scrapy.core.scraper] WARNING: Dropped: Validation failed!
{'url': 'https://www.grainger.com/product/FLIR-Videoscope-Kit-1024-×-600-797RC5', 'sku': '797RC5', 'name': None, 'supplier': 'GRAINGER'}
2024-08-01 04:51:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:302
2024-08-01 04:51:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:303
2024-08-01 04:51:26 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:304
2024-08-01 04:51:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:305
2024-08-01 04:51:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:306
2024-08-01 04:51:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:307
2024-08-01 04:51:31 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:308
2024-08-01 04:51:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:309
2024-08-01 04:51:35 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:310
2024-08-01 04:51:37 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:311
2024-08-01 04:51:38 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:312
2024-08-01 04:51:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:313
2024-08-01 04:51:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:314
2024-08-01 04:51:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:315
2024-08-01 04:51:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:316
2024-08-01 04:51:45 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:317
2024-08-01 04:51:47 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 796121 items (at 99986 items/min)
2024-08-01 04:51:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:318
2024-08-01 04:51:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:319
2024-08-01 04:51:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:320
2024-08-01 04:51:51 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:321
2024-08-01 04:51:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:322
2024-08-01 04:51:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:323
2024-08-01 04:51:55 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:324
2024-08-01 04:51:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:325
2024-08-01 04:51:58 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:326
2024-08-01 04:52:00 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:327
2024-08-01 04:52:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:328
2024-08-01 04:52:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:329
2024-08-01 04:52:04 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:330
2024-08-01 04:52:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:331
2024-08-01 04:52:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:332
2024-08-01 04:52:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:333
2024-08-01 04:52:10 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:334
2024-08-01 04:52:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:335
2024-08-01 04:52:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:336
2024-08-01 04:52:15 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:337
2024-08-01 04:52:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:338
2024-08-01 04:52:18 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:339
2024-08-01 04:52:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:340
2024-08-01 04:52:21 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:341
2024-08-01 04:52:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:342
2024-08-01 04:52:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:343
2024-08-01 04:52:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:344
2024-08-01 04:52:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:345
2024-08-01 04:52:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:346
2024-08-01 04:52:29 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:347
2024-08-01 04:52:31 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:348
2024-08-01 04:52:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:349
2024-08-01 04:52:34 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:350
2024-08-01 04:52:35 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:351
2024-08-01 04:52:37 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:352
2024-08-01 04:52:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:353
2024-08-01 04:52:40 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:354
2024-08-01 04:52:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:355
2024-08-01 04:52:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:356
2024-08-01 04:52:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:357
2024-08-01 04:52:46 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:358
2024-08-01 04:52:47 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 897369 items (at 101248 items/min)
2024-08-01 04:52:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:359
2024-08-01 04:52:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:360
2024-08-01 04:52:50 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:361
2024-08-01 04:52:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:362
2024-08-01 04:52:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:363
2024-08-01 04:52:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:364
2024-08-01 04:52:56 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:365
2024-08-01 04:52:58 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:366
2024-08-01 04:52:59 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:367
2024-08-01 04:53:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:368
2024-08-01 04:53:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:369
2024-08-01 04:53:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:370
2024-08-01 04:53:05 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:371
2024-08-01 04:53:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:372
2024-08-01 04:53:08 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:373
2024-08-01 04:53:10 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:374
2024-08-01 04:53:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:375
2024-08-01 04:53:13 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:376
2024-08-01 04:53:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:377
2024-08-01 04:53:16 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:378
2024-08-01 04:53:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:379
2024-08-01 04:53:18 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:380
2024-08-01 04:53:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:381
2024-08-01 04:53:21 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:382
2024-08-01 04:53:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:383
2024-08-01 04:53:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:384
2024-08-01 04:53:26 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:385
2024-08-01 04:53:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:386
2024-08-01 04:53:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:387
2024-08-01 04:53:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:388
2024-08-01 04:53:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:389
2024-08-01 04:53:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:390
2024-08-01 04:53:35 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:391
2024-08-01 04:53:36 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:392
2024-08-01 04:53:38 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:393
2024-08-01 04:53:40 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:394
2024-08-01 04:53:41 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:395
2024-08-01 04:53:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:396
2024-08-01 04:53:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:397
2024-08-01 04:53:46 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:398
2024-08-01 04:53:47 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 997721 items (at 100352 items/min)
2024-08-01 04:53:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:399
2024-08-01 04:53:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:400
2024-08-01 04:53:50 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:401
2024-08-01 04:53:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:402
2024-08-01 04:53:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:403
2024-08-01 04:53:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:404
2024-08-01 04:53:56 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:405
2024-08-01 04:53:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:406
2024-08-01 04:53:58 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:407
2024-08-01 04:54:00 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:408
2024-08-01 04:54:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:409
2024-08-01 04:54:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:410
2024-08-01 04:54:05 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:411
2024-08-01 04:54:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:412
2024-08-01 04:54:08 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:413
2024-08-01 04:54:10 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:414
2024-08-01 04:54:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:415
2024-08-01 04:54:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:416
2024-08-01 04:54:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:417
2024-08-01 04:54:15 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:418
2024-08-01 04:54:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:419
2024-08-01 04:54:18 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:420
2024-08-01 04:54:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:421
2024-08-01 04:54:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:422
2024-08-01 04:54:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:423
2024-08-01 04:54:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:424
2024-08-01 04:54:26 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:425
2024-08-01 04:54:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:426
2024-08-01 04:54:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:427
2024-08-01 04:54:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:428
2024-08-01 04:54:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:429
2024-08-01 04:54:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:430
2024-08-01 04:54:34 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:431
2024-08-01 04:54:36 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:432
2024-08-01 04:54:37 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:433
2024-08-01 04:54:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:434
2024-08-01 04:54:41 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:435
2024-08-01 04:54:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:436
2024-08-01 04:54:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:437
2024-08-01 04:54:45 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:438
2024-08-01 04:54:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:439
2024-08-01 04:54:47 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 1098732 items (at 101011 items/min)
2024-08-01 04:54:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:440
2024-08-01 04:54:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:441
2024-08-01 04:54:51 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:442
2024-08-01 04:54:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:443
2024-08-01 04:54:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:444
2024-08-01 04:54:55 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:445
2024-08-01 04:54:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:446
2024-08-01 04:54:58 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:447
2024-08-01 04:55:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:448
2024-08-01 04:55:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:449
2024-08-01 04:55:04 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:450
2024-08-01 04:55:04 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:451
2024-08-01 04:55:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:452
2024-08-01 04:55:08 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:453
2024-08-01 04:55:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:454
2024-08-01 04:55:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:455
2024-08-01 04:55:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:456
2024-08-01 04:55:13 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:457
2024-08-01 04:55:15 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:458
2024-08-01 04:55:16 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:459
2024-08-01 04:55:18 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:460
2024-08-01 04:55:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:461
2024-08-01 04:55:21 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:462
2024-08-01 04:55:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:463
2024-08-01 04:55:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:464
2024-08-01 04:55:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:465
2024-08-01 04:55:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:466
2024-08-01 04:55:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:467
2024-08-01 04:55:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:468
2024-08-01 04:55:31 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:469
2024-08-01 04:55:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:470
2024-08-01 04:55:34 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:471
2024-08-01 04:55:35 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:472
2024-08-01 04:55:37 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:473
2024-08-01 04:55:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:474
2024-08-01 04:55:40 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:475
2024-08-01 04:55:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:476
2024-08-01 04:55:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:477
2024-08-01 04:55:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:478
2024-08-01 04:55:46 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:479
2024-08-01 04:55:47 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 1199739 items (at 101007 items/min)
2024-08-01 04:55:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:480
2024-08-01 04:55:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:481
2024-08-01 04:55:50 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:482
2024-08-01 04:55:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:483
2024-08-01 04:55:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:484
2024-08-01 04:55:55 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:485
2024-08-01 04:55:56 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:486
2024-08-01 04:55:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:487
2024-08-01 04:55:59 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:488
2024-08-01 04:56:00 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:489
2024-08-01 04:56:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:490
2024-08-01 04:56:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:491
2024-08-01 04:56:05 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:492
2024-08-01 04:56:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:493
2024-08-01 04:56:08 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:494
2024-08-01 04:56:10 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:495
2024-08-01 04:56:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:496
2024-08-01 04:56:13 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:497
2024-08-01 04:56:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:498
2024-08-01 04:56:16 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:499
2024-08-01 04:56:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:500
2024-08-01 04:56:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:501
2024-08-01 04:56:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:502
2024-08-01 04:56:21 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:503
2024-08-01 04:56:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:504
2024-08-01 04:56:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:505
2024-08-01 04:56:26 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:506
2024-08-01 04:56:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:507
2024-08-01 04:56:29 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:508
2024-08-01 04:56:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:509
2024-08-01 04:56:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:510
2024-08-01 04:56:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:511
2024-08-01 04:56:35 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:512
2024-08-01 04:56:37 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:513
2024-08-01 04:56:38 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:514
2024-08-01 04:56:40 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:515
2024-08-01 04:56:41 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:516
2024-08-01 04:56:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:517
2024-08-01 04:56:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:518
2024-08-01 04:56:46 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:519
2024-08-01 04:56:47 [scrapy.extensions.logstats] INFO: Crawled 29 pages (at 0 pages/min), scraped 1300179 items (at 100440 items/min)
2024-08-01 04:56:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:520
2024-08-01 04:56:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:521
2024-08-01 04:56:50 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:522
2024-08-01 04:56:51 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:523
2024-08-01 04:56:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:524
2024-08-01 04:56:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:525
2024-08-01 04:56:56 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:526
2024-08-01 04:56:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:527
2024-08-01 04:56:58 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:528
2024-08-01 04:57:00 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:529
2024-08-01 04:57:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:530
2024-08-01 04:57:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:531
2024-08-01 04:57:04 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:532
2024-08-01 04:57:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:533
2024-08-01 04:57:08 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:534
2024-08-01 04:57:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:535
2024-08-01 04:57:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:536
2024-08-01 04:57:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:537
2024-08-01 04:57:13 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:538
2024-08-01 04:57:15 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:539
2024-08-01 04:57:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:540
2024-08-01 04:57:18 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:541
2024-08-01 04:57:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:542
2024-08-01 04:57:21 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:543
2024-08-01 04:57:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:544
2024-08-01 04:57:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:545
2024-08-01 04:57:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:546
2024-08-01 04:57:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:547
2024-08-01 04:57:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:548
2024-08-01 04:57:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:549
2024-08-01 04:57:31 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:550
2024-08-01 04:57:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:551
2024-08-01 04:57:34 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:552
2024-08-01 04:57:36 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:553
2024-08-01 04:57:38 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:554
2024-08-01 04:57:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:555
2024-08-01 04:57:39 [scrapy.core.engine] INFO: Closing spider (finished)
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] ------------------------------ MONITORS ------------------------------
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] Extracted Items Monitor/test_stat_monitor... OK
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] Item Validation Monitor/test_stat_monitor... FAIL
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] Error Count Monitor/test_stat_monitor... SKIPPED (Unable to find 'log_count/ERROR' in job stats.)
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] Warning Count Monitor/test_stat_monitor... FAIL
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] Finish Reason Monitor/Should have the expected finished reason(s)... OK
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] Unwanted HTTP codes monitor/Should not hit the limit of unwanted http status... OK
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] Field Coverage Monitor/test_check_if_field_coverage_rules_are_met... OK
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] Retry Count monitor/Should not hit the limit of requests that reached the maximum retry amount... OK
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] Downloader Exceptions monitor/test_stat_monitor... SKIPPED (Unable to find 'downloader/exception_count' in job stats.)
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] Successful Requests monitor/Should have at least the minimum number of successful requests... OK
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] Total Requests monitor/Should not hit the total limit of requests... OK
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] ----------------------------------------------------------------------
2024-08-01 04:57:39 [grainger] ERROR: [Spidermon] 
======================================================================
FAIL: Item Validation Monitor/test_stat_monitor
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 184, in test_stat_monitor
    assertion_method(
AssertionError: Expecting 'spidermon/validation/fields/errors' to be '<=' to '0.0'. Current value: '2'

2024-08-01 04:57:39 [grainger] ERROR: [Spidermon] 
======================================================================
FAIL: Warning Count Monitor/test_stat_monitor
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 184, in test_stat_monitor
    assertion_method(
AssertionError: Expecting 'log_count/WARNING' to be '<=' to '1.0'. Current value: '2'

2024-08-01 04:57:39 [grainger] INFO: [Spidermon] 11 monitors in 0.003s
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] FAILED (failures=2, skipped=2)
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] -------------------------- FINISHED ACTIONS --------------------------
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] ----------------------------------------------------------------------
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] 0 actions in 0.000s
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] OK
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] --------------------------- PASSED ACTIONS ---------------------------
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] ----------------------------------------------------------------------
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] 0 actions in 0.000s
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] OK
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] --------------------------- FAILED ACTIONS ---------------------------
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] ----------------------------------------------------------------------
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] 0 actions in 0.000s
2024-08-01 04:57:39 [grainger] INFO: [Spidermon] OK
2024-08-01 04:57:40 [scrapy.extensions.feedexport] INFO: Stored bq feed (786 items) in: bq://response-elt.scraper_data.catalog_urls/batch:556
2024-08-01 04:57:40 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 55938,
 'downloader/request_count': 29,
 'downloader/request_method_count/GET': 29,
 'downloader/response_bytes': 8242855,
 'downloader/response_count': 29,
 'downloader/response_status_count/200': 29,
 'elapsed_time_seconds': 832.789415,
 'feedexport/success_count/BigQueryFeedStorage': 556,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2024, 8, 1, 4, 57, 39, 913257, tzinfo=datetime.timezone.utc),
 'httpcompression/response_bytes': 8882489,
 'httpcompression/response_count': 29,
 'item_dropped_count': 2,
 'item_dropped_reasons_count/DropItem': 2,
 'item_scraped_count': 1388286,
 'log_count/ERROR': 2,
 'log_count/INFO': 607,
 'log_count/WARNING': 2,
 'memusage/max': 2964787200,
 'memusage/startup': 113094656,
 'poet/injector/catalog_discovery.pages.grainger.GraingerSitemapPageObject': 29,
 'request_depth_max': 1,
 'response_received_count': 29,
 'scheduler/dequeued': 29,
 'scheduler/dequeued/memory': 29,
 'scheduler/enqueued': 29,
 'scheduler/enqueued/memory': 29,
 'spidermon/validation/fields': 5553152,
 'spidermon/validation/fields/errors': 2,
 'spidermon/validation/fields/errors/invalid_uri': 2,
 'spidermon/validation/fields/errors/invalid_uri/url': 2,
 'spidermon/validation/items': 1388288,
 'spidermon/validation/items/dropped': 2,
 'spidermon/validation/items/errors': 2,
 'spidermon/validation/validators': 1,
 'spidermon/validation/validators/item/jsonschema': True,
 'spidermon_field_coverage/dict/name': 1.0,
 'spidermon_field_coverage/dict/sku': 1.0,
 'spidermon_field_coverage/dict/supplier': 1.0,
 'spidermon_field_coverage/dict/url': 1.0,
 'spidermon_item_scraped_count': 1388286,
 'spidermon_item_scraped_count/dict': 1388286,
 'spidermon_item_scraped_count/dict/name': 1388286,
 'spidermon_item_scraped_count/dict/sku': 1388286,
 'spidermon_item_scraped_count/dict/supplier': 1388286,
 'spidermon_item_scraped_count/dict/url': 1388286,
 'start_time': datetime.datetime(2024, 8, 1, 4, 43, 47, 123842, tzinfo=datetime.timezone.utc)}
2024-08-01 04:57:40 [scrapy.core.engine] INFO: Spider closed (finished)