CrawlSpider Link Builder’s settings can be accessed via the settings TAB
Ideally, you may never need to update any of the default settings except turning on/off the logging for troubleshooting purpose.
Allowed Article Post Types:
This is a list of available WordPress post types. Only the selected types will be considered for link discovery and internal link building.
Article Batch Limit per Crawl
The default value suggested range is between 50 to 150. This value controls the batch size of processing the number of articles each time CrawlSpider starts analyzing the content. If you have a large website with 1000+ articles, then this setting will help to control the compute resources.
Max Outbound Internal Links Per Page
This setting controls the max amount of internal links any page is supposed to give out.
Min Words to Skip for Link Inserts
This is the suggested number of words to skip from the top before looking for any relevant anchor text for internal linking.
Min number of characters for thin Pages
A thin page is any article that has only few words. In order to suggest and build links the source and target page both need to have good amount of content. This setting tells to ignore such thin pages. The number of characters under 500 is the default setting to treat the page as a thin page.
Min Anchor Text Word Count
The suggested value for this setting is 2. You can try higher setting such as 3 or 4 words but this may produce fewer link suggestions.
Max Anchor Text Word Count
This setting tells not to look for anchor text that has words more than this setting.
Background Process
Keep this value On. Without this the crawler will not do anything.
Logging Status
On
Match Quality Lower Bound
Keep this setting to 2.
Internal Link Open Target
Defalut value is ‘_blank’
External Link Open Target
Default value is ‘_blank’
Internal Link rel attribute
Default value is ‘noopener’
External Link rel attribute
Defalut value is ‘follow’
Crawl Throttle Delay (seconds)
Defalut value is 0. If you see lot of 403 forbidden errors for your own URLs then try to change this value to 5 seconds.
Auto Linking Options
Auto Links – Max Outbound Links Per Page
When using Auto Linking Rule engine, this setting will control the number of outgoing links per page.