Get indexed at all. Nothing else matters until this works.
This is a gate, not a spectrum. A page that's 95% technically perfect but blocked from crawling gets exactly the same result as a page that doesn't exist. JobPlnr right now — zero indexed pages on Google, a 401 on a basic automated homepage check — is a live example of this gate being shut.
site:jobplnr.com returns zero results. A watchdog script's first homepage check returned HTTP 401. This is very likely a bot-protection or firewall setting on Hostinger blocking non-browser requests — which blocks Googlebot exactly the same way it blocks a health-check script. Fix this before investing a single hour in the modules below; they're pointless while this gate is shut.
The three places this breaks, in order of likelihood
- Bot/WAF protection applied site-wide. hPanel → Website → Security — look for "Bot Protection" or a firewall rule that blocks unrecognized user-agents.
robots.txtdisallowing everything. Checkjobplnr.com/robots.txtdirectly. A leftoverDisallow: /from staging is extremely common.- No sitemap submitted, or Search Console never set up. Even a fully crawlable site can sit un-indexed for a long time without an explicit nudge.
robots.txt in 2026 has a second job: sorting AI crawlers
There are now two distinct families of AI bot, and treating them the same is the most common new mistake (one 2026 audit found 41% of B2B sites still block at least one major AI bot they didn't mean to):
| Bot family | What it does | Examples |
|---|---|---|
| Training crawlers | Collect content to train the underlying model. Blocking these doesn't affect whether you're cited in live answers. | GPTBot, ClaudeBot, Google-Extended, anthropic-ai |
| Retrieval / search agents | Fetch pages live to answer a specific user question and cite a source. Blocking these directly removes you from AI answers. | ChatGPT-User, OAI-SearchBot, Claude-User, PerplexityBot |
Once the bot-block is fixed: confirm robots.txt looks like the block above, submit a sitemap in Search Console, then use the URL Inspection tool on your homepage and click "Request Indexing." This can move you from zero to indexed in days instead of waiting for Google's default crawl schedule.
Deep Dive Beyond the basics — crawl budget, JS rendering, Core Web Vitals
Crawl budget — not your problem yet, but know what it is
Google allocates a finite "budget" of pages it's willing to crawl per site per day, roughly scaled to the site's authority and size. At JobPlnr's current scale (a handful of core pages, not thousands of programmatically-generated pages) this is genuinely not a constraint. It becomes relevant if you later build something like per-city or per-role landing pages at scale (e.g. "resume analysis for [role] in [city]") — at that point, a bloated or wasteful crawl (redirect chains, duplicate parameter URLs, soft-404s) can cause Google to under-crawl your actually-important pages. Worth knowing the term exists; not worth optimizing for today.
JavaScript rendering — the two-wave indexing problem
If any part of JobPlnr renders content client-side via JavaScript (React/Vue-style dynamic content, not server-rendered PHP output), Google indexes it in two passes: first it crawls the raw HTML, then — later, sometimes days later — it renders the JavaScript in a headless browser to see the final page. Content that only appears after JS execution can be indexed late or incompletely. Check this directly: Search Console → URL Inspection → "View Crawled Page" → compare the rendered screenshot against what a real visitor sees. If your core value-prop text (pricing, "how it works") only appears after JS runs and isn't in the initial server-rendered HTML, that's worth fixing — since your stack is PHP-first, this is likely a non-issue for most of the site, but worth spot-checking any React/JS-heavy sections.
Canonical tags — preventing self-inflicted duplicate content
Marketing campaigns often append tracking parameters (?utm_source=linkedin, ?ref=...) to URLs you share. Without a <link rel="canonical" href="..."> tag pointing back to the clean URL, Google can treat jobplnr.com/?utm_source=linkedin and jobplnr.com/ as two separate pages competing with each other, diluting both. Add a self-referencing canonical tag to every page's <head> pointing to its own clean URL — cheap insurance against this.
Core Web Vitals — a real, if secondary, ranking input
Google measures three real-user performance metrics as a (minor but real) ranking factor: LCP (Largest Contentful Paint — how fast the main content appears), INP (Interaction to Next Paint — how responsive the page feels when clicked/tapped), and CLS (Cumulative Layout Shift — whether content jumps around while loading). Check yours free at pagespeed.web.dev. On Hostinger shared hosting, the most common offender is unoptimized images — the hero banner work done earlier in JobPlnr's build already addressed this once; worth re-checking after any new page ships.
Sitemap hygiene
Two rules that matter more than people expect: only include URLs that actually return 200 and that you want indexed (never include redirected, noindexed, or broken URLs — each one is a small trust signal against the whole sitemap), and keep <lastmod> dates genuinely accurate rather than stamping every URL with today's date on every regeneration — Google has been known to discount sitemaps where every single URL claims to have changed simultaneously, since that pattern reads as unreliable metadata.
robots.txt isn't disallowing everything, and added explicit AI-bot rulessite:jobplnr.com starts returning results