List of URLs Tab
The “List of URLs” Tab is starting screen that greets you when you visit the CrawlSpider plugin. This section maintains complete list of all URLs on your website.
When you go to the crawl options [under settings TAB] and you say allow post types, so the custom post types, you can enable whatever you want to get scanned and crawled.
This setting tells the crawler what type of articles to scan. Any type that is not selected will be excluded from the scanning process. These initial set of links are the seed URLS from where the scanning will begin.
Scanning
After scanning each page, it identifies additional links within those pages, continuously expanding its list. Imagine your blog or website contains 50 articles, posts, and pages. As it navigates through these 50 pages, it uncovers a variety of links, including external, category, tag, and partner links. Consequently, all these links are compiled into this list, creating a comprehensive repository of both your internal and external links.
Metric Collection
The crawler operating in the background will examine every link provided, collecting data and metrics for each one. As demonstrated here, you can view the full URL, including an analysis of the domain and other pertinent information.
Internal/External
It determines if the URL is internal or external , the page size and the word counts. The background process will completely scan each URL that is a WordPress Post, Page or a custom post type.
Keyword phrases, NLP tokenization
In this process it collects all the keywords, H1 tags and each and every link on the page. The keywords are processed through a NLP algorithm that is later used in the Link Suggestion/Discovery phase.
Besides, it gathers various metrics. Each metric reveals the number of contextual links a page provides, including external, do-follow, and tag links. This allows viewing a range of metrics for every URL at once.
Navigational Links vs Contextual links
There’s a distinction between navigational and contextual links. Links within the article’s main body are contextual, while those in the navigation area, like tags and categories, are considered navigational links.
Broken Link Checks
The crawler tests every link, checking if it’s accessible, and records the received status. A 200 status code means the link is valid. Codes above 400 indicate broken or error links.
Therefore, there is a possibility that certain links might be obsolete. It will indicate if a link is unreachable or broken, and it will record this information. Additionally, it will alert you through a color-coding system.
Download and Link Audits
Hence this screen will provide you a current state of all your internal and external links. You can download the entire list as CSV for link audits and detailed SEO analysis.
Disabled URLs
You can also enable or disable any URL by clicking on the URL and then clicking on the Toggle button. Disabled URLs will not be scanned and used for link suggestion/discovery process.
List of URL metrics
OutBound Links :
Total outbound links (internal + external domains). These includes the navigational links such as menu, header and footer links as well.
OutBound Internal :
Subset of Total Outbound links that are internal only. These includes the navigational links such as menu, header and footer links as well.
OutBound Contextual Internal :
All internal links that are within the body of the content. These excludes the navigational links such as sitewide footer and header links etc.
OutBound Contextual External :
All external links that are within the body of the content. These excludes the navigational links such as sitewide footer and header links etc.
OutBound Do follow External :
All do-follow external links that are within the body of the content. These excludes the navigational links such as sitewide footer and header links etc.
OutBound Category
All the category links that show in the navigation or at the top or bottom of the content
OutBound Tag
All the category links that show in the navigation or at the top or bottom of the content
OutBound External
All External links whether they are contextual or sitewide links.
OutBound Sitewide
Total sitewide links such as the menu and navigational links
OutBound Image
Total outbound links to images that are either internal or external
InBound
Total inbound links. These includes the navigational links such as menu, header and footer links as well. This metric tells us “How many links are pointing to this page”
Other metrics similar to the Outbound are as below
- InBound Internal
- InBound Contextual Internal
- InBound Category
- InBound Tag
- InBound External
- InBound Sitewide