2024-09-27 16:24:17 [scrapy.utils.log] INFO: Scrapy 2.11.2 started (bot: catalog_discovery)
2024-09-27 16:24:17 [scrapy.utils.log] INFO: Versions: lxml 5.2.2.0, libxml2 2.12.6, cssselect 1.2.0, parsel 1.9.1, w3lib 2.1.2, Twisted 24.3.0, Python 3.11.9 (main, Aug 13 2024, 01:19:58) [GCC 12.2.0], pyOpenSSL 24.1.0 (OpenSSL 3.2.1 30 Jan 2024), cryptography 42.0.7, Platform Linux-6.4.10-dirty-x86_64-with-glibc2.36
2024-09-27 16:24:17 [scrapy.addons] INFO: Enabled addons:
[]
2024-09-27 16:24:17 [scrapy.extensions.telnet] INFO: Telnet Password: 53d59b97fa4f7e4f
2024-09-27 16:24:17 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
 'scrapy.extensions.telnet.TelnetConsole',
 'scrapy.extensions.memusage.MemoryUsage',
 'scrapy.extensions.feedexport.FeedExporter',
 'scrapy.extensions.logstats.LogStats',
 'spidermon.contrib.scrapy.extensions.Spidermon']
2024-09-27 16:24:17 [scrapy.crawler] INFO: Overridden settings:
{'BOT_NAME': 'catalog_discovery',
 'CONCURRENT_ITEMS': 1000,
 'CONCURRENT_REQUESTS': 32,
 'FEED_EXPORT_ENCODING': 'utf-8',
 'LOG_FILE': '/var/lib/scrapyd/logs/catalog_discovery/lowes/ecafde667cec11efa9cf4200a9fe0102.log',
 'LOG_LEVEL': 'INFO',
 'NEWSPIDER_MODULE': 'catalog_discovery.spiders',
 'REQUEST_FINGERPRINTER_CLASS': 'scrapy_poet.ScrapyPoetRequestFingerprinter',
 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7',
 'RETRY_TIMES': 5,
 'SPIDER_MODULES': ['catalog_discovery.spiders'],
 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor',
 'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:125.0) '
               'Gecko/20100101 Firefox/125.0'}
2024-09-27 16:24:17 [scrapy_poet.injection] INFO: Loading providers:
 [<class 'scrapy_poet.page_input_providers.HttpRequestProvider'>,
 <class 'scrapy_poet.page_input_providers.HttpResponseProvider'>,
 <class 'scrapy_poet.page_input_providers.HttpClientProvider'>,
 <class 'scrapy_poet.page_input_providers.PageParamsProvider'>,
 <class 'scrapy_poet.page_input_providers.RequestUrlProvider'>,
 <class 'scrapy_poet.page_input_providers.ResponseUrlProvider'>,
 <class 'scrapy_poet.page_input_providers.StatsProvider'>]
2024-09-27 16:24:17 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware',
 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
 'scrapy_poet.InjectionMiddleware',
 'scrapy.downloadermiddlewares.retry.RetryMiddleware',
 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
 'scrapy_poet.DownloaderStatsMiddleware']
2024-09-27 16:24:17 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
 'scrapy_poet.RetryMiddleware',
 'scrapy.spidermiddlewares.referer.RefererMiddleware',
 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
 'scrapy.spidermiddlewares.depth.DepthMiddleware']
2024-09-27 16:24:17 [scrapy.middleware] INFO: Enabled item pipelines:
['scraping_utils.pipelines.DuplicatesFilterPipeline',
 'scraping_utils.pipelines.AttachSupplierPipeline',
 'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline']
2024-09-27 16:24:17 [scrapy.core.engine] INFO: Spider opened
2024-09-27 16:24:17 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2024-09-27 16:24:17 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6024
2024-09-27 16:24:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:1
2024-09-27 16:24:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:2
2024-09-27 16:24:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:3
2024-09-27 16:24:34 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:4
2024-09-27 16:24:36 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:5
2024-09-27 16:24:37 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:6
2024-09-27 16:24:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:7
2024-09-27 16:24:41 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:8
2024-09-27 16:24:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:9
2024-09-27 16:24:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:10
2024-09-27 16:24:46 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:11
2024-09-27 16:24:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:12
2024-09-27 16:24:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:13
2024-09-27 16:24:51 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:14
2024-09-27 16:24:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:15
2024-09-27 16:24:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:16
2024-09-27 16:24:56 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:17
2024-09-27 16:24:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:18
2024-09-27 16:24:59 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:19
2024-09-27 16:25:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:20
2024-09-27 16:25:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:21
2024-09-27 16:25:05 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:22
2024-09-27 16:25:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:23
2024-09-27 16:25:08 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:24
2024-09-27 16:25:10 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:25
2024-09-27 16:25:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:26
2024-09-27 16:25:13 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:27
2024-09-27 16:25:15 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:28
2024-09-27 16:25:16 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:29
2024-09-27 16:25:17 [scrapy.extensions.logstats] INFO: Crawled 33 pages (at 33 pages/min), scraped 75231 items (at 75231 items/min)
2024-09-27 16:25:18 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:30
2024-09-27 16:25:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:31
2024-09-27 16:25:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:32
2024-09-27 16:25:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:33
2024-09-27 16:25:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:34
2024-09-27 16:25:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:35
2024-09-27 16:25:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:36
2024-09-27 16:25:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:37
2024-09-27 16:25:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:38
2024-09-27 16:25:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:39
2024-09-27 16:25:35 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:40
2024-09-27 16:25:37 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:41
2024-09-27 16:25:38 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:42
2024-09-27 16:25:40 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:43
2024-09-27 16:25:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:44
2024-09-27 16:25:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:45
2024-09-27 16:25:45 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:46
2024-09-27 16:25:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:47
2024-09-27 16:25:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:48
2024-09-27 16:25:50 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:49
2024-09-27 16:25:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:50
2024-09-27 16:25:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:51
2024-09-27 16:25:55 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:52
2024-09-27 16:25:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:53
2024-09-27 16:25:59 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:54
2024-09-27 16:26:00 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:55
2024-09-27 16:26:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:56
2024-09-27 16:26:04 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:57
2024-09-27 16:26:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:58
2024-09-27 16:26:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:59
2024-09-27 16:26:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:60
2024-09-27 16:26:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:61
2024-09-27 16:26:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:62
2024-09-27 16:26:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:63
2024-09-27 16:26:16 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:64
2024-09-27 16:26:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:65
2024-09-27 16:26:17 [scrapy.extensions.logstats] INFO: Crawled 33 pages (at 0 pages/min), scraped 163455 items (at 88224 items/min)
2024-09-27 16:26:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:66
2024-09-27 16:26:21 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:67
2024-09-27 16:26:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:68
2024-09-27 16:26:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:69
2024-09-27 16:26:26 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:70
2024-09-27 16:26:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:71
2024-09-27 16:26:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:72
2024-09-27 16:26:31 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:73
2024-09-27 16:26:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:74
2024-09-27 16:26:35 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:75
2024-09-27 16:26:36 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:76
2024-09-27 16:26:38 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:77
2024-09-27 16:26:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:78
2024-09-27 16:26:41 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:79
2024-09-27 16:26:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:80
2024-09-27 16:26:45 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:81
2024-09-27 16:26:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:82
2024-09-27 16:26:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:83
2024-09-27 16:26:50 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:84
2024-09-27 16:26:51 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:85
2024-09-27 16:26:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:86
2024-09-27 16:26:55 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:87
2024-09-27 16:26:56 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:88
2024-09-27 16:26:58 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:89
2024-09-27 16:27:00 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:90
2024-09-27 16:27:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:91
2024-09-27 16:27:04 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:92
2024-09-27 16:27:05 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:93
2024-09-27 16:27:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:94
2024-09-27 16:27:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:95
2024-09-27 16:27:10 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:96
2024-09-27 16:27:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:97
2024-09-27 16:27:13 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:98
2024-09-27 16:27:15 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:99
2024-09-27 16:27:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:100
2024-09-27 16:27:17 [scrapy.extensions.logstats] INFO: Crawled 33 pages (at 0 pages/min), scraped 251511 items (at 88056 items/min)
2024-09-27 16:27:18 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:101
2024-09-27 16:27:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:102
2024-09-27 16:27:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:103
2024-09-27 16:27:30 [scrapy.core.scraper] WARNING: Dropped: Validation failed!
{'url': 'https://www.lowes.com/pd/origin 21-28-in-L-x-28-in-W-Round-Gold-Framed-Wall-Mirror/1000421755', 'sku': None, 'name': None, 'supplier': 'LOWES'}
2024-09-27 16:27:31 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:104
2024-09-27 16:27:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:105
2024-09-27 16:27:35 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:106
2024-09-27 16:27:37 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:107
2024-09-27 16:27:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:108
2024-09-27 16:27:40 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:109
2024-09-27 16:27:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:110
2024-09-27 16:27:44 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:111
2024-09-27 16:27:45 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:112
2024-09-27 16:27:47 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:113
2024-09-27 16:27:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:114
2024-09-27 16:27:50 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:115
2024-09-27 16:27:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:116
2024-09-27 16:27:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:117
2024-09-27 16:27:55 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:118
2024-09-27 16:27:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:119
2024-09-27 16:27:59 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:120
2024-09-27 16:28:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:121
2024-09-27 16:28:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:122
2024-09-27 16:28:04 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:123
2024-09-27 16:28:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:124
2024-09-27 16:28:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:125
2024-09-27 16:28:19 [scrapy.extensions.logstats] INFO: Crawled 66 pages (at 33 pages/min), scraped 314605 items (at 63094 items/min)
2024-09-27 16:28:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:126
2024-09-27 16:28:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:127
2024-09-27 16:28:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:128
2024-09-27 16:28:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:129
2024-09-27 16:28:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:130
2024-09-27 16:28:29 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:131
2024-09-27 16:28:31 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:132
2024-09-27 16:28:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:133
2024-09-27 16:28:34 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:134
2024-09-27 16:28:36 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:135
2024-09-27 16:28:38 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:136
2024-09-27 16:28:39 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:137
2024-09-27 16:28:41 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:138
2024-09-27 16:28:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:139
2024-09-27 16:28:45 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:140
2024-09-27 16:28:46 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:141
2024-09-27 16:28:48 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:142
2024-09-27 16:28:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:143
2024-09-27 16:28:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:144
2024-09-27 16:28:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:145
2024-09-27 16:28:55 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:146
2024-09-27 16:28:57 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:147
2024-09-27 16:28:58 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:148
2024-09-27 16:29:00 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:149
2024-09-27 16:29:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:150
2024-09-27 16:29:03 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:151
2024-09-27 16:29:05 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:152
2024-09-27 16:29:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:153
2024-09-27 16:29:08 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:154
2024-09-27 16:29:10 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:155
2024-09-27 16:29:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:156
2024-09-27 16:29:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:157
2024-09-27 16:29:16 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:158
2024-09-27 16:29:17 [scrapy.extensions.logstats] INFO: Crawled 66 pages (at 0 pages/min), scraped 397773 items (at 83168 items/min)
2024-09-27 16:29:18 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:159
2024-09-27 16:29:20 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:160
2024-09-27 16:29:21 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:161
2024-09-27 16:29:23 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:162
2024-09-27 16:29:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:163
2024-09-27 16:29:26 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:164
2024-09-27 16:29:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:165
2024-09-27 16:29:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:166
2024-09-27 16:29:32 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:167
2024-09-27 16:29:34 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:168
2024-09-27 16:29:36 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:169
2024-09-27 16:29:46 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:170
2024-09-27 16:29:50 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:171
2024-09-27 16:29:52 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:172
2024-09-27 16:29:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:173
2024-09-27 16:30:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:174
2024-09-27 16:30:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:175
2024-09-27 16:30:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:176
2024-09-27 16:30:17 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:177
2024-09-27 16:30:17 [scrapy.extensions.logstats] INFO: Crawled 66 pages (at 0 pages/min), scraped 443724 items (at 45951 items/min)
2024-09-27 16:30:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:178
2024-09-27 16:30:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:179
2024-09-27 16:31:01 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:180
2024-09-27 16:31:02 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:181
2024-09-27 16:31:21 [scrapy.extensions.logstats] INFO: Crawled 66 pages (at 0 pages/min), scraped 455040 items (at 11316 items/min)
2024-09-27 16:31:43 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:182
2024-09-27 16:32:14 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:183
2024-09-27 16:32:18 [scrapy.extensions.logstats] INFO: Crawled 66 pages (at 0 pages/min), scraped 460042 items (at 5002 items/min)
2024-09-27 16:32:19 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:184
2024-09-27 16:33:24 [scrapy.extensions.logstats] INFO: Crawled 66 pages (at 0 pages/min), scraped 461526 items (at 1484 items/min)
2024-09-27 16:34:19 [scrapy.extensions.logstats] INFO: Crawled 66 pages (at 0 pages/min), scraped 462552 items (at 1026 items/min)
2024-09-27 16:34:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:185
2024-09-27 16:34:42 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:186
2024-09-27 16:35:05 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:187
2024-09-27 16:35:07 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:188
2024-09-27 16:35:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:189
2024-09-27 16:35:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:190
2024-09-27 16:35:17 [scrapy.extensions.logstats] INFO: Crawled 66 pages (at 0 pages/min), scraped 477479 items (at 14927 items/min)
2024-09-27 16:35:22 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:191
2024-09-27 16:35:25 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:192
2024-09-27 16:35:27 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:193
2024-09-27 16:35:29 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:194
2024-09-27 16:35:50 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:195
2024-09-27 16:36:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:196
2024-09-27 16:36:18 [scrapy.extensions.logstats] INFO: Crawled 66 pages (at 0 pages/min), scraped 493095 items (at 15616 items/min)
2024-09-27 16:36:18 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:197
2024-09-27 16:36:28 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:198
2024-09-27 16:36:30 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:199
2024-09-27 16:36:33 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:200
2024-09-27 16:37:19 [scrapy.extensions.logstats] INFO: Crawled 68 pages (at 2 pages/min), scraped 500036 items (at 6941 items/min)
2024-09-27 16:38:20 [scrapy.extensions.logstats] INFO: Crawled 69 pages (at 1 pages/min), scraped 500044 items (at 8 items/min)
2024-09-27 16:39:26 [scrapy.extensions.logstats] INFO: Crawled 71 pages (at 2 pages/min), scraped 500055 items (at 11 items/min)
2024-09-27 16:40:26 [scrapy.extensions.logstats] INFO: Crawled 71 pages (at 0 pages/min), scraped 500072 items (at 17 items/min)
2024-09-27 16:41:17 [scrapy.extensions.logstats] INFO: Crawled 75 pages (at 4 pages/min), scraped 501207 items (at 1135 items/min)
2024-09-27 16:42:22 [scrapy.extensions.logstats] INFO: Crawled 79 pages (at 4 pages/min), scraped 501547 items (at 340 items/min)
2024-09-27 16:43:24 [scrapy.extensions.logstats] INFO: Crawled 79 pages (at 0 pages/min), scraped 501599 items (at 52 items/min)
2024-09-27 16:44:19 [scrapy.extensions.logstats] INFO: Crawled 79 pages (at 0 pages/min), scraped 501644 items (at 45 items/min)
2024-09-27 16:45:09 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:201
2024-09-27 16:45:20 [scrapy.extensions.logstats] INFO: Crawled 82 pages (at 3 pages/min), scraped 504220 items (at 2576 items/min)
2024-09-27 16:45:24 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:202
2024-09-27 16:46:11 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:203
2024-09-27 16:46:12 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:204
2024-09-27 16:46:28 [scrapy.extensions.logstats] INFO: Crawled 82 pages (at 0 pages/min), scraped 511772 items (at 7552 items/min)
2024-09-27 16:46:46 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:205
2024-09-27 16:46:49 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:206
2024-09-27 16:46:51 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:207
2024-09-27 16:46:53 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:208
2024-09-27 16:46:54 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:209
2024-09-27 16:47:06 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:210
2024-09-27 16:47:18 [scrapy.extensions.logstats] INFO: Crawled 82 pages (at 0 pages/min), scraped 527516 items (at 15744 items/min)
2024-09-27 16:47:26 [scrapy.extensions.feedexport] INFO: Stored bq feed (2500 items) in: bq://response-elt.scraper_data.catalog_urls/batch:211
2024-09-27 16:48:26 [scrapy.extensions.logstats] INFO: Crawled 82 pages (at 0 pages/min), scraped 530038 items (at 2522 items/min)
2024-09-27 16:49:21 [scrapy.extensions.logstats] INFO: Crawled 82 pages (at 0 pages/min), scraped 530268 items (at 230 items/min)
2024-09-27 16:50:20 [scrapy.extensions.logstats] INFO: Crawled 82 pages (at 0 pages/min), scraped 530342 items (at 74 items/min)
2024-09-27 16:51:26 [scrapy.extensions.logstats] INFO: Crawled 82 pages (at 0 pages/min), scraped 531308 items (at 966 items/min)
2024-09-27 16:52:19 [scrapy.extensions.logstats] INFO: Crawled 82 pages (at 0 pages/min), scraped 531398 items (at 90 items/min)
2024-09-27 16:53:27 [scrapy.extensions.logstats] INFO: Crawled 82 pages (at 0 pages/min), scraped 531548 items (at 150 items/min)
2024-09-27 16:54:34 [scrapy.extensions.logstats] INFO: Crawled 82 pages (at 0 pages/min), scraped 531654 items (at 106 items/min)
2024-09-27 16:55:20 [scrapy.extensions.logstats] INFO: Crawled 82 pages (at 0 pages/min), scraped 531708 items (at 54 items/min)