How does CrawlSpider handle empty anchor text?
How are malformed or invalid URLs handled?
To demonstrate, the following invalid URL was introduced to the below article snippet
Once the CrawlSpider completes scanning the article, it will identify the invalid or missing URL and capture the details as below
It assigns a dummy URL to avoid spending any CPU cycles for other checks.
The URL is also listed in the list of URLs tab as below
Once you fix the anchor text with valid URL, you may delete the invalid URLs from the List of Domains and URLs view.