SEO Monitoring Tool, How to track content changes

Introduction to Content Monitoring, SEO Audit and Change Tracking

This is going to be a brief introduction to CrawlSpider SEO Monitoring tool

Initially I wanted to release this SEO Monitor tool as a cloud based service but then decided to do it first as a WordPress plugin. There are several reasons to release this as a WordPress plugin and it would be good for another article by itself.

But first let me talk little bit about what this tool does and how it can help you to do website change monitor

What is the Purpose of this tool?

CrawlSpider’s SEO Audit and Change tracking tool is designed to monitor all your important pages and keep track of all the relevant SEO elements. The important SEO elements are those that help you rank your pages well on search engines. The page change monitor is a SEO Audit tool with built in change and Diff algorithm. For every page that ranks well on Google and other search engines, the key important SEO elements foundational to ranking and bringing organic traffic are Page Titles, H1 Tag, H2 Tag and all the important keywords that are sprinkled throughout the body of the page.

SEO Change Monitor will alert you as soon as it detects a change in any of the SEO structures. for e.g Let say you have a large team of web developers and when someone in your team did not follow instructions or someone submitted an accidental release of new website changes. Whether it is intentional or accidental, any website content change could have drastic effects on the search engine rankings.

The SEO of a web change is broken when say the Page title is modified or completely disappears. What if internal links get broken or if the anchor text keywords are modified or when they start pointing to different locations. What if important keywords are replaced or when the page body text is repositioned. All of these changes are worthy of your prompt attention. Don’t you think?

This tool will provide an extra set of eyes and diligently monitor all your websites and important pages for changes Day in and Day out.

What does it really monitor?

The checks are basically divided into domain level checks and page level checks.

Monitor SSL Certificate Expiration

At the DOMAIN LEVEL one of the checks is ssl monitor that checks for changes in SSL certificates, validy and expiry of SSL certs. The change monitor performs this check only for the root domain and checks for the validity and also the expiration date. For e.g it will alert you say 30 days before the SSL is about to expire. Since it has change tracking and diff algorithm, it will also alert you if any fields within your SSL certificate changes.

{"name":"\/CN=infocaptor.com","subject":{"CN":"infocaptor.com"},"hash":"..ee04..","issuer":{"C":"US","O":"Let's Encrypt","CN":"R3"},"version":2,"serialNumber":".....29563376464732","validFrom":"210807134509Z","validTo":"211105134507Z","validFrom_time_t":1628343909,"validTo_time_t":1636119907,"signatureTypeSN":"RSA-SHA256","signatureTypeLN":"sha256WithRSAEncryption","signatureTypeNID":668,"purposes":{"1":[true,false,"sslclient"],"2":[true,false,"sslserver"],"3":[true,false,"nssslserver"],"4":[false,false,"smimesign"],"5":[false,false,"smimeencrypt"],"6":[false,false,"crlsign"],"7":[true,true,"any"],"8":[true,false,"ocsphelper"],"9":[false,false,"timestampsign"]},"extensions":{"keyUsage":"Digital Signature, Key Encipherment","extendedKeyUsage":"TLS Web Server Authentication, TLS Web Client Authentication","basicConstraints":"CA:FALSE","subjectKeyIdentifier":"93:D1:30:5B:42:62:C7:27:DA:09:E9:84:B7:B9:D1:AB:0B:E7:B8:20","authorityKeyIdentifier":"keyid:14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6
","authorityInfoAccess":"OCSP - URI:http:\/\/r3.o.lencr.org
CA Issuers - URI:http:\/\/r3.i.lencr.org\/
","subjectAltName":"DNS:infocaptor.com, DNS:www.infocaptor.com","certificatePolicies":"Policy: 2.23...
Policy: 1.3.6.1.4.1.xxxx.1
  CPS: http:\/\/cps.letsencrypt.org
","ct_precert_scts":"Signed Certificate Timestamp:
    Version   : v1(0)
    Log ID    : 94:20:BC:1E:8E:D5:8D:6C:88:73:1F:82:8B:22:2C:0D:
                D1:DA:4D:5E:6C:4F:94:3D:61:DB:4E:2F:58:4D:A2:C2
    Timestamp : Aug  7 14:45:09.150 2021 GMT
    Extensions: none
    Signature : ecdsa-with-SHA256
                30:44:02:20:04:D3:48:F2:B8:37:EF:8E:2B:06:C5:80:
                3F:51:8C:02:71:BD:61:33:E1:35:27:04:0C:C3:67:A2:
                2E:07:42:A8:02:20:20:90:D7:63:E5:17:F6:44:24:95:
                C7:DD:22:D5:CC:18:16:45:1D:66:A7:05:A7:A2:10:73:
                BF:81:DE:FC:2F:C9
Signed Certificate Timestamp:
    Version   : v1(0)
    Log ID    : F6:5C:94:2F:D1:77:30:22:14:54:18:08:30:94:56:8E:
                E3:4D:13:19:33:BF:DF:0C:2F:20:0B:CC:4E:F1:64:E3
    Timestamp : Aug  7 14:45:09.149 2021 GMT
    Extensions: none
    Signature : ecdsa-with-SHA256
                30:45:02:21:00:F1:23:53:61:BC:7D:41:5E:82:BE:84:
                8E:30:0D:0B:54:0E:9C:47:6E:CB:33:EB:8E:E0:BA:CC:
                B7:9D:1E:77:D8:02:20:45:31:96:69:B9:1C:AD:5D:C2:
                CD:E1:AA:A8:7A:F4:FA:D0:87:05:52:BA:44:23:F5:A0:
                48:F3:C4:D4:B3:2B:D5"},"expire_date":"2021-11-05"}

The diff algorithm will detect change to any portion of the SSL certificate and will start alerting as you get close to the expiry date.

Another important root level check is IP address monitor. The DNS Monitor check will scan and grab the entire DNS record for the domain and keep track for any changes including any MX records.

[{"host":"mydomain.com","class":"IN","type":"A","ip":"195.16.xx.xx","target":"na"},
{"host":"mydomain.com","class":"IN","type":"MX","pri":5,"target":"alt1.aspmx.l.google.com"},
{"host":"mydomain.com","class":"IN","type":"MX","pri":5,"target":"alt2.aspmx.l.google.com"},
{"host":"mydomain.com","class":"IN","type":"MX","pri":10,"target":"alt3.aspmx.l.google.com"},
{"host":"mydomain.com","class":"IN","type":"MX","pri":1,"target":"aspmx.l.google.com"},
{"host":"mydomain.com","class":"IN","type":"NS","target":"ns10.dnsprovider.com"},
{"host":"mydomain.com","class":"IN","type":"NS","target":"ns11.dnsprovider.com"},
{"host":"mydomain.com","class":"IN","type":"NS","target":"ns12.dnsprovider.com"},
{"host":"mydomain.com","class":"IN","type":"NS","target":"ns13.dnsprovider.com"},
{"host":"mydomain.com","class":"IN","type":"SOA","mname":"ns10.dnsprovider.com"
........
{"host":"mydomain.com","class":"IN","type":"TXT"
,"txt":"google-site-verification=g12....."
,"entries":["google-site-verification=g12......"],"target":"na"}]

If and when your IP or DNS record changes it will bring your prompt attention towards it. Other domain level checks are to make sure that your website safely redirects from http to https, non-www to www or vice versa. It also checks for robots.txt validity and changes, sitemap changes and so on.

Page Change Monitoring Checks

At the PAGE Level there is a comprehensive list of tags that it tracks such as

  • Title tag
  • H1 and H2 Tag
  • It tracks all Internal links and their anchors
  • It also tracks all External links and their anchors

In addition to the default checks, you can also define custom checks using XPATH, Regex, Html Tags and CSS Class names.

Malware Detection

Although CrawlSpider is not meant to be used for malware scanning or detection purpose but during a recent testing it helped detect malware on this website and it warrants a separate article or case study. When a website gets infected with a malware that changes link anchor text for manipulating Google rankings, this is easily detected using CrawlSpider and provides as an early signal of a hacked website. This capability along with security plugins like WordFence, GotMLS or service like MalCare can provide solid defense against known virus or malware attacks.

Now lets talk a little bit about the benefits. What’s in it for you?  

The main benefit is to protect your Organic traffic and site rankings. You have worked hard and invested so much in building the organic traffic to your website … so why not protect your SEO investment, right?   

Agencies can Monitor all client domains

The tool does not limit you to monitor only your own domains. If you are running an agency, you could potentially setup this tool to monitor all of your clients websites. This way you could be the first person to inform the client that something is broken.

Isn’t that a great benefit to have?

It simply means more trust, more referrals and more business.

Words with arrows pointing to a funnel Free Photo
Use SEO monitor to generate leads

Another side benefit is, you can also set it up as an effective lead generation process. Let say you want to target your local Chiropractors or Dental websites. Rather than sending cold emails, you can input all of your potential lead’s websites into the tool and if something changes you will be the first to communicate them about the broken changes. Isn’t that a better way to start a conversation with your potential clients?

Now let us spend some time into the actual UI

The tool is delivered as a WordPress plugin, you simply upload the zip file and activate it. The scans and alerting is all auto scheduled. The only task you need to do is input a list of urls that you need to monitor.

This is the screen where you input and track all your URLs

Bulk URL monitoring

You can copy paste the urls on separate lines and hit this button.

The remaining screens are for you to define custom checks. You can also disable any of the default checks in these screens.

Custom Checks
Default Checks
Alerts

This Alert tab is where you will see all the latest Alerts. Every time a scan runs, it may generate one or many alerts.

A summary of alerts is also sent to you as an email

So this pretty much wraps up the introduction. I hope you like our SEO Monitor and Change Tracking plugin and do let us know.