Cites Google
Crawl Errors Keeping Nashville Law Firm Pages Out of Google
- 6 sections
- 1 table
- 4 sources cited
When a new practice page does not show up in Google, the cause falls into one of two groups, and they need different fixes. The first group is true crawl errors: Google tried to fetch the page and failed, because of a server error, a broken redirect or a missing page. The second group is indexing decisions: Google could reach the page, or was told not to index it, or chose another URL instead. Search Console’s Page indexing report names both groups with exact labels, and the label tells you which fix applies (Google’s Page indexing report help).
Group 1: Google Could Not Fetch the Page
These are the statuses in the Page indexing report where crawling itself went wrong:
- Server error (5xx). Google’s definition: “Your server returned a 500-level error when the page was requested.” The report notes that “server errors can be transient,” so test the URL again before rebuilding anything.
- Redirect error. Google lists “A redirect chain that was too long,” “A redirect loop,” a redirect URL that exceeded the maximum length, and “A bad or empty URL in the redirect chain.” Old practice pages that were redirected more than once during site redesigns are the place to look.
- Not found (404). “This page returned a 404 error when requested.” Fine for a page that was removed on purpose; a problem if it is a practice page or a page other sites link to.
- Soft 404. Google’s report describes a page that “returns a user-friendly ‘not found’ message but not a 404 HTTP response code.” A removed practice page that shows a friendly “page not found” message with a success code lands here.
Group 2: Google Reached the Page but Did Not Index It
These statuses are not crawl errors. Each comes from an instruction on the site or a decision by Google:
| Status in the report | What Google says it means | What to do |
|---|---|---|
| URL marked 'noindex' | The page tells Google not to index it | Remove the tag if the page should rank |
| URL blocked by robots.txt | A robots.txt rule stops crawling | Change the rule if the page should be crawled |
| Duplicate, Google chose different canonical than user | "Google thinks another URL makes a better canonical" | Check whether the two pages are really different |
| Alternate page with proper canonical tag | The page points to another URL as its main version | Nothing, if that is intended |
| Discovered – currently not indexed | "The page was found by Google, but not crawled yet" | Wait for a few pages; investigate if it covers much of the site |
| Crawled – currently not indexed | "The page was crawled by Google but not indexed" | Improve the page and how the site links to it |
Two points from Google’s documentation change how the first two rows should be handled.
robots.txt does not keep a page out of Google. Google’s robots.txt guide says: “To keep a web page out of Google, block indexing with noindex or password-protect the page” (Google’s robots.txt introduction). A URL blocked by robots.txt can still show in results, which is why the report has a separate status, “Indexed, though blocked by robots.txt.”
A noindex tag only works if Google can read it. Google’s noindex documentation says: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file” (Google on blocking indexing). Blocking a page with robots.txt and tagging it noindex at the same time prevents Google from seeing the tag.
Reading “Crawled – currently not indexed”
This label means crawling worked. Google fetched the page and did not add it to the index, and the help page does not give a reason. When it appears on a practice page, the useful questions are about the page and its place on the site:
- Does the page answer something other pages on the site do not, or is it a near-copy of another practice page or a city variant?
- Do other pages on the site link to it in their text, or only the menu?
- Is the page thin: a short description, no process, no fees, no Tennessee-specific detail?
What a practice page needs to contain is covered in why practice area pages fail to rank, and how a page gains links from the rest of the site in internal linking on Nashville law firm sites.
Crawl Budget: Google’s Guide Is Written for Much Larger Sites
Google’s crawl budget guide says it is intended mainly for “Large sites (1 million+ unique pages) with content that changes moderately often (once a week),” “Medium or larger sites (10,000+ unique pages) with very rapidly changing content (daily),” and sites with a large share of URLs listed as “Discovered – currently not indexed.” It adds: “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide” (Google’s crawl budget guide).
A law firm site with a few hundred pages is far below those sizes. Thin, near-identical city-and-service pages are still worth fixing or consolidating, but the reason is that they compete with each other and give readers little, not that they use up a crawl budget. That problem is covered in content cannibalization on law firm sites and multi-location SEO without duplicates.
A Check Before Each Practice Page Goes Live
For example, imagine a “Nashville Car Accident Lawyer” page that was built on a staging site and copied to the live site with its staging settings intact. Six weeks later it shows as “URL marked ‘noindex’.” A short check at launch would have caught it:
- View the page source and confirm there is no noindex robots meta tag.
- Confirm no robots.txt rule covers the page’s path.
- Confirm the canonical tag points to the page itself.
- Run the URL through Search Console’s URL Inspection live test; Google’s help says the live test “fetches and examines the URL in real time,” though it notes the live test “does not cover all possible indexing conditions.”
- Link to the page from at least one related page on the site, not only the menu.
Order of Work When Several Pages Are Flagged
Fix group 1 errors on practice pages first, because the page cannot be indexed at all until Google can fetch it. Next, remove stray noindex tags and wrong canonicals on pages that should rank; those fixes are quick and unambiguous. Then work through “Crawled – currently not indexed” practice pages by improving their content and internal links. “Discovered – currently not indexed” on a handful of low-value URLs can wait; Google’s description is that it “wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl.”