Robots.txt, sitemap, and HTTPS: a small-site technical checklist

HTTPS everywhere

Serve marketing and content pages over HTTPS with valid certificates. Redirect HTTP to HTTPS consistently so analytics, canonicals, and backlinks consolidate on one scheme.

IntenseSEO reports non-HTTPS URLs and mixed-content hints where detectable from the initial HTML fetch.

robots.txt sanity

robots.txt should not accidentally disallow your entire site or critical paths like /blog or product folders. Use it to steer crawl budget, not as a substitute for noindex on private pages.

Remember: robots disallow does not equal “hidden from Google” if other sites link to the URL.

XML sitemap

A sitemap helps discovery for new or large sites. It should list canonical URLs, return 200, and stay reasonably fresh after publishes.

We check reachability and basic validity signals—not whether Google will index every listed URL.

Canonical tags

Each important page should declare its preferred URL. Conflicting canonicals across duplicates dilute signals and can split reporting in Search Console.

Re-audit after changes

Technical fixes can take time to reflect in search results. Re-run a free audit after deployments to confirm the HTML we fetch matches what you expect.