Gary Illyes, a Google Analyst, recently warned in a LinkedIn post that if Google starts aggressively crawling your website, it might indicate underlying issues rather than being a purely positive sign.
Increased Crawling: A Double-Edged Sword
While more crawling can sometimes mean Google is finding your site valuable, Illyes advises caution. He mentions two common problems associated with increased crawling: infinite spaces and website hacks.
Infinite Spaces: A Common Culprit
Illyes highlights that websites with “infinite spaces” like calendar modules or endlessly filterable product listings can create unlimited URLs. This can cause a spike in crawling activity.
“If your site generally has pages that search users find helpful, crawlers will get excited about these infinite spaces for a time,” Illyes explains.
To prevent this, he recommends using the robots.txt file to block crawlers from accessing these areas.
Hacked Sites: Another Cause for Concern
A sudden increase in crawling can also indicate a security breach where hackers inject spam into your site. Initially, crawlers may treat this as new content before recognizing it as malicious.
Illyes states, “If a no-good-doer somehow managed to get access…they might flood your site with, well, crap… crawlers will get excited about these new pages for a time and happily crawl them.”
Be Skeptical of Crawling Spikes
Illyes advises treating unexpected increases in crawling as potential problems until the root cause is identified. He notes, “Treat unexpected sharp increases in crawling as a symptom…until you can prove otherwise.”
Fixing Hacked Sites: Google’s Guidance
For those dealing with hacked sites, Illyes refers to a helpful page and video from Google outlining the recovery process.
Key Steps in Google’s Recovery Process
- Identify the Vulnerability: The first step is finding how the hacker gained access. Tools like Google’s Webmaster Tools can help detect issues.
- Fix the Vulnerability: Once identified, close the security hole to prevent future unauthorized access. This might involve updating software or changing passwords.
- Clean the Hacked Content: Remove any spam, malware, or other malicious content. Security plugins like Wordfence can assist in this process.
- Harden Security: Take additional measures to secure the site, such as enabling firewalls, limiting user permissions, and regularly updating software.
- Request a Review: After cleaning the site and fixing vulnerabilities, request Google to review the site to remove any security warnings or blacklists.
The review process is typically faster for malware issues than for spam issues, as spam cleanup requires more thorough inspection.
Additional Tips from Google’s John Mueller
John Mueller has also offered advice for recovering from the SEO impact of hacked pages:
- Use the URL removal tool to quickly deindex hacked pages.
- Improve the overall quality of the site beyond just removing hacked content.
- Understand that lingering impacts may persist for months as the site works to regain Google’s trust.
By following these guidelines and being vigilant about sudden changes in crawling activity, you can better protect your site from potential issues and maintain a healthy SEO profile.
