List of URLs Tab
The “List of URLs” Tab is starting screen that greets you when you visit the CrawlSpider plugin. This section maintains complete list of all URLs on your website.
When you go to the crawl options [under settings TAB] and you say allow post types, so the custom post types, you can enable whatever you want to get scanned and crawled.
This setting tells the crawler what type of articles to scan. Any type that is not selected will be excluded from the scanning process. These initial set of links are the seed URLS from where the scanning will begin.
Scanning
After scanning each page, it identifies additional links within those pages, continuously expanding its list. Imagine your blog or website contains 50 articles, posts, and pages. As it navigates through these 50 pages, it uncovers a variety of links, including external, category, tag, and partner links. Consequently, all these links are compiled into this list, creating a comprehensive repository of both your internal and external links.
Metric Collection
The crawler operating in the background will examine every link provided, collecting data and metrics for each one. As demonstrated here, you can view the full URL, including an analysis of the domain and other pertinent information.
Internal/External
It determines if the URL is internal or external , the page size and the word counts. The background process will completely scan each URL that is a WordPress Post, Page or a custom post type.
Keyword phrases, NLP tokenization
In this process it collects all the keywords, H1 tags and each and every link on the page. The keywords are processed through a NLP algorithm that is later used in the Link Suggestion/Discovery phase.
Besides, it gathers various metrics. Each metric reveals the number of contextual links a page provides, including external, do-follow, and tag links. This allows viewing a range of metrics for every URL at once.
Navigational Links vs Contextual links
There’s a distinction between navigational and contextual links. Links within the article’s main body are contextual, while those in the navigation area, like tags and categories, are considered navigational links.
Broken Link Checks
The crawler tests every link, checking if it’s accessible, and records the received status. A 200 status code means the link is valid. Codes above 400 indicate broken or error links.
Therefore, there is a possibility that certain links might be obsolete. It will indicate if a link is unreachable or broken, and it will record this information. Additionally, it will alert you through a color-coding system.
Download and Link Audits
Hence this screen will provide you a current state of all your internal and external links. You can download the entire list as CSV for link audits and detailed SEO analysis.
Disabled URLs
You can also enable or disable any URL by clicking on the URL and then clicking on the Toggle button. Disabled URLs will not be scanned and used for link suggestion/discovery process.
List of URL metrics
[TODO : Document each metric ]