2025-01-21 21:27:57 [scrapy.utils.log] (PID: 285) INFO: Scrapy 2.11.2 started (bot: catalog_extraction) 2025-01-21 21:27:57 [scrapy.utils.log] (PID: 285) INFO: Versions: lxml 5.2.2.0, libxml2 2.12.6, cssselect 1.2.0, parsel 1.9.1, w3lib 2.1.2, Twisted 24.3.0, Python 3.11.11 (main, Dec 4 2024, 20:38:25) [GCC 12.2.0], pyOpenSSL 24.1.0 (OpenSSL 3.2.1 30 Jan 2024), cryptography 42.0.7, Platform Linux-6.4.10-dirty-x86_64-with-glibc2.36 2025-01-21 21:27:57 [wb_mason] (PID: 285) INFO: Starting extraction spider wb_mason... 2025-01-21 21:27:57 [scrapy.addons] (PID: 285) INFO: Enabled addons: [] 2025-01-21 21:27:57 [scrapy.extensions.telnet] (PID: 285) INFO: Telnet Password: cb0d94899c17de6a 2025-01-21 21:27:57 [scrapy.middleware] (PID: 285) INFO: Enabled extensions: ['scrapy.extensions.corestats.CoreStats', 'scrapy.extensions.telnet.TelnetConsole', 'scrapy.extensions.closespider.CloseSpider', 'scrapy.extensions.feedexport.FeedExporter', 'scrapy.extensions.logstats.LogStats', 'scrapy_playwright.memusage.ScrapyPlaywrightMemoryUsageExtension', 'spidermon.contrib.scrapy.extensions.Spidermon'] 2025-01-21 21:27:57 [scrapy.crawler] (PID: 285) INFO: Overridden settings: {'BOT_NAME': 'catalog_extraction', 'CONCURRENT_ITEMS': 250, 'CONCURRENT_REQUESTS': 24, 'DOWNLOAD_DELAY': 1.25, 'FEED_EXPORT_ENCODING': 'utf-8', 'HTTPPROXY_ENABLED': False, 'LOG_FILE': '/var/lib/scrapyd/logs/catalog_extraction/wb_mason/93c9735ed83e11efba394200a9fe0102.log', 'LOG_FORMAT': '%(asctime)s [%(name)s] (PID: %(process)d) %(levelname)s: ' '%(message)s', 'LOG_LEVEL': 'INFO', 'NEWSPIDER_MODULE': 'catalog_extraction.spiders', 'REQUEST_FINGERPRINTER_CLASS': 'scrapy_poet.ScrapyPoetRequestFingerprinter', 'REQUEST_FINGERPRINTER_IMPLEMENTATION': '2.7', 'RETRY_TIMES': 5, 'SPIDER_MODULES': ['catalog_extraction.spiders'], 'TWISTED_REACTOR': 'twisted.internet.asyncioreactor.AsyncioSelectorReactor', 'USER_AGENT': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, ' 'like Gecko) Chrome/129.0.0.0 Safari/537.36'} 2025-01-21 21:27:58 [scrapy-playwright] (PID: 285) WARNING: Connecting to remote browser, ignoring PLAYWRIGHT_LAUNCH_OPTIONS 2025-01-21 21:27:58 [scrapy-playwright] (PID: 285) WARNING: Connecting to remote browser, ignoring PLAYWRIGHT_LAUNCH_OPTIONS 2025-01-21 21:27:58 [scrapy_poet.injection] (PID: 285) INFO: Loading providers: [, , , , , , ] 2025-01-21 21:27:58 [scrapy.middleware] (PID: 285) INFO: Enabled downloader middlewares: ['scrapy.downloadermiddlewares.offsite.OffsiteMiddleware', 'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware', 'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware', 'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware', 'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware', 'scrapy_poet.InjectionMiddleware', 'scrapy.downloadermiddlewares.retry.RetryMiddleware', 'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware', 'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware', 'scrapy.downloadermiddlewares.redirect.RedirectMiddleware', 'scrapy.downloadermiddlewares.cookies.CookiesMiddleware', 'scrapy_poet.DownloaderStatsMiddleware'] 2025-01-21 21:27:58 [wb_mason] (PID: 285) WARNING: Missing 'PARSING_ERRORS_STORAGE'. Middleware will NOT store the HTML of pages with parsing errors. 2025-01-21 21:27:58 [scrapy.middleware] (PID: 285) INFO: Enabled spider middlewares: ['catalog_extraction.middlewares.ErrorHandlerSpiderMiddleware', 'catalog_extraction.middlewares.FixtureSavingMiddleware', 'scrapy_poet.RetryMiddleware', 'scrapy.spidermiddlewares.referer.RefererMiddleware', 'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware', 'scrapy.spidermiddlewares.depth.DepthMiddleware'] 2025-01-21 21:27:58 [scrapy.middleware] (PID: 285) INFO: Enabled item pipelines: ['catalog_extraction.pipelines.DuplicatedSKUsFilterPipeline', 'catalog_extraction.pipelines.DiscontinuedProductsAdjustmentPipeline', 'catalog_extraction.pipelines.PriceRoundingPipeline', 'scraping_utils.pipelines.AttachSupplierPipeline', 'spidermon.contrib.scrapy.pipelines.ItemValidationPipeline'] 2025-01-21 21:27:58 [scrapy.core.engine] (PID: 285) INFO: Spider opened 2025-01-21 21:27:58 [scrapy.extensions.closespider] (PID: 285) INFO: Spider will stop when no items are produced after 7200 seconds. 2025-01-21 21:27:58 [scrapy.extensions.logstats] (PID: 285) INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min) 2025-01-21 21:27:58 [scrapy.extensions.telnet] (PID: 285) INFO: Telnet console listening on 127.0.0.1:6023 2025-01-21 21:27:58 [scrapy-playwright] (PID: 285) INFO: Starting download handler 2025-01-21 21:27:58 [scrapy-playwright] (PID: 285) INFO: Starting download handler 2025-01-21 21:28:03 [wb_mason] (PID: 285) INFO: Batch query job with batch size 100 and offset 0 2025-01-21 21:28:04 [scrapy-playwright] (PID: 285) INFO: Connecting using CDP: https://brd-customer-hl_13cda1e4-zone-main_scraping_browser:l9p73ctebkrc@brd.superproxy.io:9222 2025-01-21 21:28:05 [scrapy-playwright] (PID: 285) INFO: Connected using CDP: https://brd-customer-hl_13cda1e4-zone-main_scraping_browser:l9p73ctebkrc@brd.superproxy.io:9222 2025-01-21 21:28:06 [scrapy-playwright] (PID: 285) WARNING: Closing page due to failed request: exc_type= exc_msg=Page.goto: Protocol error (Page.navigate): Missing Credentials (missing_credentials) Call log: navigating to "https://www.wbmason.com/DynamicPage.aspx?Page=WB-Custom-Products", waiting until "load" Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Missing Credentials (missing_credentials) Call log: navigating to "https://www.wbmason.com/DynamicPage.aspx?Page=WB-Custom-Products", waiting until "load" 2025-01-21 21:28:06 [scrapy.core.scraper] (PID: 285) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks result = context.run( File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 54, in process_request return (yield download_func(request=request, spider=spider)) File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1251, in adapt extracted: _SelfResultT | Failure = result.result() File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 383, in _download_request return await self._download_request_with_retry(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Missing Credentials (missing_credentials) Call log: navigating to "https://www.wbmason.com/DynamicPage.aspx?Page=WB-Custom-Products", waiting until "load" 2025-01-21 21:28:10 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:10 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:11 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:12 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:14 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:15 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:18 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:18 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:20 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:21 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:23 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:25 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:27 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:28 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:30 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:31 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:33 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:34 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:36 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:38 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:40 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:45 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:28:58 [scrapy.extensions.logstats] (PID: 285) INFO: Crawled 22 pages (at 22 pages/min), scraped 0 items (at 0 items/min) 2025-01-21 21:29:04 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:29:04 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:29:04 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:29:25 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:29:26 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:29:28 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:29:43 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:29:48 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:29:49 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:29:57 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:29:58 [scrapy.extensions.logstats] (PID: 285) INFO: Crawled 32 pages (at 10 pages/min), scraped 0 items (at 0 items/min) 2025-01-21 21:29:59 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:30:06 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:30:19 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:30:22 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:30:26 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:30:33 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:30:37 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:30:47 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:30:54 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:30:58 [scrapy.extensions.logstats] (PID: 285) INFO: Crawled 41 pages (at 9 pages/min), scraped 0 items (at 0 items/min) 2025-01-21 21:31:00 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:31:05 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:31:21 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:31:26 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:31:32 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:31:35 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:31:41 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:31:45 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:31:46 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:31:52 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:31:56 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:31:58 [scrapy.extensions.logstats] (PID: 285) INFO: Crawled 52 pages (at 11 pages/min), scraped 0 items (at 0 items/min) 2025-01-21 21:31:59 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:32:08 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:32:12 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:32:19 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:32:25 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:32:30 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:32:34 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:32:39 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:32:44 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:32:48 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:32:50 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:32:58 [scrapy.extensions.logstats] (PID: 285) INFO: Crawled 63 pages (at 11 pages/min), scraped 0 items (at 0 items/min) 2025-01-21 21:32:58 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:02 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:07 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:14 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:14 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:17 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:25 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:29 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:31 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:34 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:39 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:42 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:45 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:47 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:50 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:52 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:53 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:54 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:54 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:56 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:57 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:57 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:58 [scrapy.extensions.logstats] (PID: 285) INFO: Crawled 85 pages (at 22 pages/min), scraped 0 items (at 0 items/min) 2025-01-21 21:33:59 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:59 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:33:59 [scrapy.core.scraper] (PID: 285) ERROR: Spider error processing (referer: None) Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/defer.py", line 295, in aiter_errback yield await it.__anext__() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 374, in __anext__ return await self.data.__anext__() ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy/utils/python.py", line 355, in _async_chain async for o in as_async_generator(it): File "/usr/local/lib/python3.11/site-packages/scrapy/utils/asyncgen.py", line 14, in as_async_generator async for r in it: File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/referer.py", line 355, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/urllength.py", line 30, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/usr/local/lib/python3.11/site-packages/scrapy/spidermiddlewares/depth.py", line 35, in process_spider_output_async async for r in result or (): File "/usr/local/lib/python3.11/site-packages/scrapy/core/spidermw.py", line 118, in process_async async for r in iterable: File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/spiders/__init__.py", line 158, in parse_product yield await page.to_item() ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 85, in _to_item validation_item = self._validate_input() ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/utils.py", line 205, in inner return cached_meth(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/web_poet/pages.py", line 144, in _validate_input validation_item = self.validate_input() # type: ignore[attr-defined] ^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 18, in validate_input super().validate_input() File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/__init__.py", line 32, in validate_input if not self.is_product_page(): ^^^^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 21, in is_product_page if not self.get_product_json: ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/functools.py", line 1001, in __get__ val = self.func(instance) ^^^^^^^^^^^^^^^^^^^ File "/var/lib/scrapyd/eggs/catalog_extraction/1737494857.egg/catalog_extraction/pages/wb_mason.py", line 29, in get_product_json return json.loads( ^^^^^^^^^^^ File "/usr/local/lib/python3.11/json/__init__.py", line 339, in loads raise TypeError(f'the JSON object must be str, bytes or bytearray, ' TypeError: the JSON object must be str, bytes or bytearray, not NoneType 2025-01-21 21:34:01 [scrapy-playwright] (PID: 285) WARNING: Closing page due to failed request: exc_type= exc_msg=Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-Tops-QP", waiting until "load" Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-Tops-QP", waiting until "load" 2025-01-21 21:34:01 [scrapy-playwright] (PID: 285) WARNING: Closing page due to failed request: exc_type= exc_msg=Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-Tork", waiting until "load" Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-Tork", waiting until "load" 2025-01-21 21:34:01 [scrapy-playwright] (PID: 285) WARNING: Closing page due to failed request: exc_type= exc_msg=Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-Tripp-Lite", waiting until "load" Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-Tripp-Lite", waiting until "load" 2025-01-21 21:34:01 [scrapy.core.scraper] (PID: 285) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks result = context.run( File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 54, in process_request return (yield download_func(request=request, spider=spider)) File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1251, in adapt extracted: _SelfResultT | Failure = result.result() File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 383, in _download_request return await self._download_request_with_retry(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-Tops-QP", waiting until "load" 2025-01-21 21:34:02 [scrapy.core.scraper] (PID: 285) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks result = context.run( File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 54, in process_request return (yield download_func(request=request, spider=spider)) File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1251, in adapt extracted: _SelfResultT | Failure = result.result() File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 383, in _download_request return await self._download_request_with_retry(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-Tork", waiting until "load" 2025-01-21 21:34:02 [scrapy.core.scraper] (PID: 285) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks result = context.run( File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 54, in process_request return (yield download_func(request=request, spider=spider)) File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1251, in adapt extracted: _SelfResultT | Failure = result.result() File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 383, in _download_request return await self._download_request_with_retry(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-Tripp-Lite", waiting until "load" 2025-01-21 21:34:03 [scrapy-playwright] (PID: 285) WARNING: Closing page due to failed request: exc_type= exc_msg=Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-Varidesk", waiting until "load" Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-Varidesk", waiting until "load" 2025-01-21 21:34:03 [scrapy.core.scraper] (PID: 285) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks result = context.run( File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 54, in process_request return (yield download_func(request=request, spider=spider)) File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1251, in adapt extracted: _SelfResultT | Failure = result.result() File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 383, in _download_request return await self._download_request_with_retry(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-Varidesk", waiting until "load" 2025-01-21 21:34:03 [scrapy-playwright] (PID: 285) WARNING: Closing page due to failed request: exc_type= exc_msg=Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-whattabargain", waiting until "load" Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-whattabargain", waiting until "load" 2025-01-21 21:34:03 [scrapy.core.scraper] (PID: 285) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks result = context.run( File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 54, in process_request return (yield download_func(request=request, spider=spider)) File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1251, in adapt extracted: _SelfResultT | Failure = result.result() File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 383, in _download_request return await self._download_request_with_retry(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-whattabargain", waiting until "load" 2025-01-21 21:34:05 [scrapy-playwright] (PID: 285) WARNING: Closing page due to failed request: exc_type= exc_msg=Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-whattabargain-locations", waiting until "load" Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-whattabargain-locations", waiting until "load" 2025-01-21 21:34:05 [scrapy.core.scraper] (PID: 285) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks result = context.run( File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 54, in process_request return (yield download_func(request=request, spider=spider)) File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1251, in adapt extracted: _SelfResultT | Failure = result.result() File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 383, in _download_request return await self._download_request_with_retry(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-whattabargain-locations", waiting until "load" 2025-01-21 21:34:06 [scrapy-playwright] (PID: 285) WARNING: Closing page due to failed request: exc_type= exc_msg=Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-wide-format-printing", waiting until "load" Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-wide-format-printing", waiting until "load" 2025-01-21 21:34:06 [scrapy.core.scraper] (PID: 285) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks result = context.run( File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 54, in process_request return (yield download_func(request=request, spider=spider)) File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1251, in adapt extracted: _SelfResultT | Failure = result.result() File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 383, in _download_request return await self._download_request_with_retry(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=WB-wide-format-printing", waiting until "load" 2025-01-21 21:34:08 [scrapy-playwright] (PID: 285) WARNING: Closing page due to failed request: exc_type= exc_msg=Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=breakroom-supplies", waiting until "load" Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=breakroom-supplies", waiting until "load" 2025-01-21 21:34:08 [scrapy.core.scraper] (PID: 285) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks result = context.run( File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 54, in process_request return (yield download_func(request=request, spider=spider)) File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1251, in adapt extracted: _SelfResultT | Failure = result.result() File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 383, in _download_request return await self._download_request_with_retry(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/LandingPage.aspx?LandingPageID=breakroom-supplies", waiting until "load" 2025-01-21 21:34:09 [scrapy-playwright] (PID: 285) WARNING: Closing page due to failed request: exc_type= exc_msg=Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/ProductDetail.aspx?ItemDesc=1850-Coffee-Black-Gold-Dark-Roast-Ground-12-oz-Bag&ItemID=FOLG60516EA&uom=EA", waiting until "load" Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/ProductDetail.aspx?ItemDesc=1850-Coffee-Black-Gold-Dark-Roast-Ground-12-oz-Bag&ItemID=FOLG60516EA&uom=EA", waiting until "load" 2025-01-21 21:34:09 [scrapy.core.scraper] (PID: 285) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks result = context.run( File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 54, in process_request return (yield download_func(request=request, spider=spider)) File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1251, in adapt extracted: _SelfResultT | Failure = result.result() File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 383, in _download_request return await self._download_request_with_retry(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/ProductDetail.aspx?ItemDesc=1850-Coffee-Black-Gold-Dark-Roast-Ground-12-oz-Bag&ItemID=FOLG60516EA&uom=EA", waiting until "load" 2025-01-21 21:34:11 [scrapy-playwright] (PID: 285) WARNING: Closing page due to failed request: exc_type= exc_msg=Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/ProductDetail.aspx?ItemDesc=1850-Coffee-Black-Gold-Dark-Roast-Ground-12-oz-Bag-6-CT&ItemID=FOLG60516&uom=CT", waiting until "load" Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/ProductDetail.aspx?ItemDesc=1850-Coffee-Black-Gold-Dark-Roast-Ground-12-oz-Bag-6-CT&ItemID=FOLG60516&uom=CT", waiting until "load" 2025-01-21 21:34:11 [scrapy.core.scraper] (PID: 285) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks result = context.run( File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 54, in process_request return (yield download_func(request=request, spider=spider)) File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1251, in adapt extracted: _SelfResultT | Failure = result.result() File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 383, in _download_request return await self._download_request_with_retry(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/ProductDetail.aspx?ItemDesc=1850-Coffee-Black-Gold-Dark-Roast-Ground-12-oz-Bag-6-CT&ItemID=FOLG60516&uom=CT", waiting until "load" 2025-01-21 21:34:13 [scrapy-playwright] (PID: 285) WARNING: Closing page due to failed request: exc_type= exc_msg=Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/ProductDetail.aspx?ItemDesc=1850-Coffee-Black-Gold-Dark-Roast-Whole-Bean-12-oz-Bag&ItemID=FOL60518&uom=EA", waiting until "load" Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/ProductDetail.aspx?ItemDesc=1850-Coffee-Black-Gold-Dark-Roast-Whole-Bean-12-oz-Bag&ItemID=FOL60518&uom=EA", waiting until "load" 2025-01-21 21:34:13 [scrapy.core.scraper] (PID: 285) ERROR: Error downloading Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1999, in _inlineCallbacks result = context.run( File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 519, in throwExceptionIntoGenerator return g.throw(self.value.with_traceback(self.tb)) File "/usr/local/lib/python3.11/site-packages/scrapy/core/downloader/middleware.py", line 54, in process_request return (yield download_func(request=request, spider=spider)) File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1251, in adapt extracted: _SelfResultT | Failure = result.result() File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 383, in _download_request return await self._download_request_with_retry(request=request, spider=spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 436, in _download_request_with_retry return await self._download_request_with_page(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 465, in _download_request_with_page response, download = await self._get_response_and_download(request, page, spider) File "/usr/local/lib/python3.11/site-packages/scrapy_playwright/handler.py", line 565, in _get_response_and_download response = await page.goto(url=request.url, **page_goto_kwargs) File "/usr/local/lib/python3.11/site-packages/playwright/async_api/_generated.py", line 8657, in goto await self._impl_obj.goto( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_page.py", line 519, in goto return await self._main_frame.goto(**locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_frame.py", line 145, in goto await self._channel.send("goto", locals_to_params(locals())) File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 59, in send return await self._connection.wrap_api_call( File "/usr/local/lib/python3.11/site-packages/playwright/_impl/_connection.py", line 514, in wrap_api_call raise rewrite_error(error, f"{parsed_st['apiName']}: {error}") from None playwright._impl._errors.Error: Page.goto: Protocol error (Page.navigate): Unexpected Status 502 (no_peers_cooling) (proxy_no_peers_cooling) Call log: navigating to "https://www.wbmason.com/ProductDetail.aspx?ItemDesc=1850-Coffee-Black-Gold-Dark-Roast-Whole-Bean-12-oz-Bag&ItemID=FOL60518&uom=EA", waiting until "load" 2025-01-21 21:34:13 [scrapy.core.engine] (PID: 285) INFO: Closing spider (finished) 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] ------------------------------ MONITORS ------------------------------ 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] Extracted Items Monitor/test_stat_monitor... FAIL 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] Item Validation Monitor/test_stat_monitor... SKIPPED (Unable to find 'spidermon/validation/fields/errors' in job stats.) 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] Error Count Monitor/test_stat_monitor... FAIL 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] Warning Count Monitor/test_stat_monitor... FAIL 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] Finish Reason Monitor/Should have the expected finished reason(s)... OK 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] Unwanted HTTP codes monitor/Should not hit the limit of unwanted http status... OK 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] Field Coverage Monitor/test_check_if_field_coverage_rules_are_met... FAIL 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] Retry Count monitor/Should not hit the limit of requests that reached the maximum retry amount... OK 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] Downloader Exceptions monitor/test_stat_monitor... OK 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] Successful Requests monitor/Should have at least the minimum number of successful requests... OK 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] Total Requests monitor/Should not hit the total limit of requests... OK 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] ---------------------------------------------------------------------- 2025-01-21 21:34:13 [wb_mason] (PID: 285) ERROR: [Spidermon] ====================================================================== FAIL: Extracted Items Monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 177, in test_stat_monitor self.fail(message) AssertionError: Unable to find 'item_scraped_count' in job stats. 2025-01-21 21:34:13 [wb_mason] (PID: 285) ERROR: [Spidermon] ====================================================================== FAIL: Error Count Monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 184, in test_stat_monitor assertion_method( AssertionError: Expecting 'log_count/ERROR' to be '<=' to '0.0'. Current value: '100' 2025-01-21 21:34:13 [wb_mason] (PID: 285) ERROR: [Spidermon] ====================================================================== FAIL: Warning Count Monitor/test_stat_monitor ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/base.py", line 184, in test_stat_monitor assertion_method( AssertionError: Expecting 'log_count/WARNING' to be '<=' to '1.0'. Current value: '15' 2025-01-21 21:34:13 [wb_mason] (PID: 285) ERROR: [Spidermon] ====================================================================== FAIL: Field Coverage Monitor/test_check_if_field_coverage_rules_are_met ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/spidermon/contrib/scrapy/monitors/monitors.py", line 476, in test_check_if_field_coverage_rules_are_met self.assertTrue(len(failures) == 0, msg=msg) AssertionError: The following items did not meet field coverage rules: dict/inStock (expected 1.0, got 0) dict/name (expected 1.0, got 0) dict/prices (expected 1.0, got 0) dict/productStatus (expected 1.0, got 0) dict/supplier (expected 1.0, got 0) dict/supplierSku (expected 1.0, got 0) dict/url (expected 1.0, got 0) 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] 11 monitors in 0.003s 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] FAILED (failures=4, skipped=1) 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] -------------------------- FINISHED ACTIONS -------------------------- 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] ---------------------------------------------------------------------- 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] 0 actions in 0.000s 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] OK 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] --------------------------- PASSED ACTIONS --------------------------- 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] ---------------------------------------------------------------------- 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] 0 actions in 0.000s 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] OK 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] --------------------------- FAILED ACTIONS --------------------------- 2025-01-21 21:34:13 [spidermon.contrib.actions.slack] (PID: 285) INFO: :skull: `wb_mason` *spider finished with errors!* _(errors=4)_ 2025-01-21 21:34:13 [spidermon.contrib.actions.slack] (PID: 285) INFO: [ { "text": "• _Extracted Items Monitor/test_stat_monitor_: Unable to find 'item_scraped_count' in job stats.\n• _Error Count Monitor/test_stat_monitor_: Expecting 'log_count/ERROR' to be '<=' to '0.0'. Current value: '100'\n• _Warning Count Monitor/test_stat_monitor_: Expecting 'log_count/WARNING' to be '<=' to '1.0'. Current value: '15'\n• _Field Coverage Monitor/test_check_if_field_coverage_rules_are_met_: The following items did not meet field coverage rules: dict/inStock (expected 1.0, got 0) dict/name (expected 1.0, got 0) dict/prices (expected 1.0, got 0) dict/productStatus (expected 1.0, got 0) dict/supplier (expected 1.0, got 0) dict/supplierSku (expected 1.0, got 0) dict/url (expected 1.0, got 0)\n", "color": "danger", "mrkdwn_in": ["text", "pretext"] } , ] 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] CustomTemplateSendSlackMessageSpiderFinished... OK 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] ---------------------------------------------------------------------- 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] 1 action in 0.012s 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: [Spidermon] OK 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: 88 URLs returned invalid responses. 2025-01-21 21:34:13 [wb_mason] (PID: 285) INFO: Stored 88 Invalid Response URLs for later retry in Big Query. 2025-01-21 21:34:13 [scrapy.extensions.feedexport] (PID: 285) INFO: No data to insert into BigQuery - closing feed storage 2025-01-21 21:34:13 [scrapy.extensions.feedexport] (PID: 285) INFO: Stored bq feed (0 items) in: bq://response-elt.dev_scrapers.catalog_item_scrape/batch:1 2025-01-21 21:34:13 [scrapy.statscollectors] (PID: 285) INFO: Dumping Scrapy stats: {'ErrorHandlerSpiderMiddleware/TypeError': 352, 'ErrorHandlerSpiderMiddleware/invalid_responses/table/stored': 88, 'ErrorHandlerSpiderMiddleware/invalid_responses/table/to_store': 88, 'big_query/url': 100, 'downloader/exception_count': 12, 'downloader/exception_type_count/playwright._impl._errors.Error': 12, 'downloader/request_bytes': 32757, 'downloader/request_count': 100, 'downloader/request_method_count/GET': 100, 'downloader/response_bytes': 7316110, 'downloader/response_count': 88, 'downloader/response_status_count/200': 88, 'elapsed_time_seconds': 375.080482, 'feedexport/success_count/BigQueryFeedStorage': 1, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2025, 1, 21, 21, 34, 13, 411227, tzinfo=datetime.timezone.utc), 'log_count/ERROR': 104, 'log_count/INFO': 57, 'log_count/WARNING': 15, 'memusage/max': 340905984, 'memusage/startup': 127651840, 'playwright/browser_count': 1, 'playwright/context_count': 1, 'playwright/context_count/max_concurrent': 1, 'playwright/context_count/persistent/False': 1, 'playwright/context_count/remote/True': 1, 'playwright/page_count': 100, 'playwright/page_count/closed': 100, 'playwright/page_count/max_concurrent': 3, 'playwright/request_count': 7596, 'playwright/request_count/aborted': 1754, 'playwright/request_count/method/GET': 7314, 'playwright/request_count/method/POST': 282, 'playwright/request_count/navigation': 712, 'playwright/request_count/resource_type/document': 712, 'playwright/request_count/resource_type/fetch': 36, 'playwright/request_count/resource_type/font': 282, 'playwright/request_count/resource_type/image': 1295, 'playwright/request_count/resource_type/other': 56, 'playwright/request_count/resource_type/ping': 80, 'playwright/request_count/resource_type/script': 3144, 'playwright/request_count/resource_type/stylesheet': 945, 'playwright/request_count/resource_type/xhr': 1046, 'playwright/response_count': 3823, 'playwright/response_count/method/GET': 3672, 'playwright/response_count/method/POST': 151, 'playwright/response_count/resource_type/document': 389, 'playwright/response_count/resource_type/font': 263, 'playwright/response_count/resource_type/image': 107, 'playwright/response_count/resource_type/script': 1958, 'playwright/response_count/resource_type/stylesheet': 891, 'playwright/response_count/resource_type/xhr': 215, 'poet/injector/catalog_extraction.pages.wb_mason.WBMasonPageObject': 88, 'response_received_count': 88, 'scheduler/dequeued': 100, 'scheduler/dequeued/memory': 100, 'scheduler/enqueued': 100, 'scheduler/enqueued/memory': 100, 'spider_exceptions/TypeError': 88, 'spidermon/validation/validators': 1, 'spidermon/validation/validators/item/jsonschema': True, 'start_requests/big_query': 100, 'start_time': datetime.datetime(2025, 1, 21, 21, 27, 58, 330745, tzinfo=datetime.timezone.utc)} 2025-01-21 21:34:13 [scrapy.core.engine] (PID: 285) INFO: Spider closed (finished) 2025-01-21 21:34:13 [scrapy-playwright] (PID: 285) INFO: Closing download handler 2025-01-21 21:34:13 [scrapy-playwright] (PID: 285) INFO: Closing download handler 2025-01-21 21:34:13 [scrapy-playwright] (PID: 285) INFO: Closing browser