Google Analyst Gary Illyes recently shared a LinkedIn post highlighting issues web crawlers face: soft 404 errors and other hidden errors. These errors can harm your SEO efforts.
What Are Soft 404 Errors?
Soft 404 errors happen when a web server wrongly returns a “200 OK” status for pages that don’t exist or have error messages. This confuses web crawlers, making them waste time and resources on non-existent or unhelpful content.
Illyes compared it to visiting a coffee shop where nothing on the menu is actually available. This is frustrating for people and problematic for web crawlers. As Illyes said:
“Crawlers use status codes to understand if a page fetch was successful, even if the page just shows an error message. They might keep revisiting the same page, wasting resources. If there are many such pages, it wastes even more resources.”
The Hidden Costs of Soft Errors
Soft 404 errors do more than just waste crawler resources. According to Illyes, these pages rarely show up in search results because they are filtered out during indexing.
To fix this, Illyes suggests using the correct HTTP status codes when errors occur. This helps crawlers understand the situation and use their resources more efficiently. He also warned against limiting crawler rates with messages like “TOO MANY REQUESTS SLOW DOWN” because crawlers can’t understand such text-based instructions.
Why It Matters
Soft 404 errors can affect a website’s ability to be crawled and indexed. By fixing these issues, crawlers can focus on fetching and indexing valuable content, potentially boosting the site’s visibility in search results. Removing soft 404 errors also makes server resource use more efficient, as crawlers won’t waste bandwidth on error pages.
How to Fix Soft 404 Errors
To identify and fix soft 404 errors on your website, follow these steps:
- Regularly check your website’s crawl reports and logs to find pages returning “200 OK” status codes despite having error messages.
- Ensure proper error handling on your server, serving error pages with the correct HTTP status codes (e.g., 404 for not found, 410 for permanently removed).
- Use tools like Google Search Console to monitor your site’s coverage and find any pages flagged as soft 404 errors.
By addressing soft 404 errors, you can improve your website’s crawlability, indexing, and SEO performance.
