Difference in the number of records between crawls
Problem
A noticeable increase or decrease in the number of records between crawls may be due to:- Temporary site problems may prevent the crawler from finding everything.
-
Changes to the site. If your site changed but the crawler configuration wasnāt, the crawl might not fully capture the site content.
For example, if your companyās job listings page moves from
www.example-company.com/careerstowww.work-at-example-company.com, the crawl wouldnāt include the new job listing URLs. - Timing issues with site updates and crawls. For example, you update your siteās content at 8Ā AM every day, but the crawler schedule is 9Ā AM. If the updates arenāt completed on time, the crawl may miss the new content, and users might come across outdated information.
Solutions
| Issue | Solution |
|---|---|
| Temporary site problems | Perform a manual re-crawl once the site issues are resolved |
| Changes to the site | Update your crawler configuration as needed and perform a re-crawl. For large changes, consider updating the safety check threshold |
| Timing issues | Allow ample time for updates to complete before the crawler schedule |