thelastapi
Console
Products

Search

Every production deploy sends the site's new and changed pages to IndexNow, and with the account's own Google Search Console connected each site is verified, its sitemap submitted and its indexed pages and searches read every day.

Open as .md

Search gets a business's pages in front of search engines without anyone remembering to do it. Console ▸ SEO shows every site at once (pick a domain for its full view), and each project has the same view under ▸ SEO, vibiz search, the vibiz_search tool or GET /v1/projects/:id/search.

After every deploy: IndexNow

When a production deploy goes live (a build, a promote or a rollback), thelastapi reads the site's /sitemap.xml (every child sitemap of a sitemap index too) and sends the pages that are new, or whose <lastmod> changed, to IndexNow. IndexNow hands them to Bing, Yandex, Naver and Seznam, usually within minutes. A site with no sitemap sends its home page once.

  • Nothing to set up. IndexNow proves the site is yours with a key file at /<key>.txt. Every production deployment gets it, in public/ for apps with a package.json (or static/ when the app already has one), at the root for a plain static site, or inside the one folder a built site is served from (like site/, holding its index.html and its sitemap or robots.txt) when there's no homepage at the root. Google's and Bing's proof files go to the same place. An app that ships its own file at that path keeps it.
  • Own domain only. Pages are sent for the project's first live domain. The free <slug>.vibiz.app address is not sent unless the owner turns on Also send the free vibiz.app address (vibiz search free-address on): half-built projects shouldn't get indexed under the shared domain, and a business that moves to its own domain later starts clean. Once a domain is live, only the domain is sent.
  • Only what changed. Each page is sent once, then again when its lastmod changes. A new domain starts the list over.
  • Send now (vibiz search submit, POST /v1/projects/:id/search/submit) sends without a deploy.

When a run sends nothing, note says why: no own domain yet, or the key file isn't served yet (the first deploy after Search arrived adds it; deploy again).

Google doesn't take IndexNow: it reads the sitemap, and the account's own Google Search Console (below) is how it hears about the site.

Google Search Console

Add a Google key once, in console ▸ Settings ▸ Search (or vibiz search connect google --key key.json --owner-email you@gmail.com, or POST /v1/search/google/key with { key, owner_email? }). Nobody signs in with Google:

  1. In Google Cloud, pick or create a project and turn on the Search Console API and the Site Verification API.
  2. IAM ▸ Service accounts ▸ Create service account (any name, no roles), then Keys ▸ Add key ▸ Create new key ▸ JSON. Paste or choose that file.
  3. Optionally give your own Google email: it is added as an owner of every site, so they also appear in your own Search Console.

The key is checked with Google before it is kept (sealed, never shown again), and an API that is off is named with the link to turn it on. A key holds up to 1,000 sites; add another one when it fills up. Only the account owner adds keys.

Or sign in with Google instead (Or sign in with Google, POST /v1/search/google/connect), when the Stack has a Google sign-in client set up.

From then on, every project with a live own domain (or the free address, when it is sent) is set up in your Search Console, with no step of yours:

  1. Proved yours. A domain bought through thelastapi is proved with a DNS record we add to its zone, and becomes a Domain property (sc-domain:example.com, covering www and every subdomain). Any other domain, and the free address, is proved with Google's verification file (/google<code>.html), added to every production deployment the same way as IndexNow's key; it is verified a minute after the next production deploy goes live, and the file stays on later deploys, as Google checks again now and then.
  2. Added to Search Console as a property of the connected Google account.
  3. Sitemap submitted: https://<domain>/sitemap.xml, once the site serves one.
  4. Every day: clicks, impressions and average position by day, the top 50 searches and the top 50 pages over 28 days; and up to 200 pages a day inspected (URL Inspection): pages never checked first, then pages whose lastmod changed since, then the longest ago. Each gets Google's verdict and words (Submitted and indexed, Crawled - currently not indexed, Discovered - currently not indexed…), the last crawl, and the canonical Google chose.

The project's Search tab (and GET /v1/projects/:id/search, vibiz search) shows indexed and not-indexed pages, clicks, impressions and average position, a 28-day chart, the pages table and the top searches and pages. Mission Control's board shows 42 · 31 idx in the Pages cell (sitemap pages · indexed) and a Search clicks 7d column.

Push to Google (opt-in, off by default). Turn on Push pages to Google in Settings ▸ Search (PATCH /v1/search/settings { "google_push": true }, vibiz search push on) and every day the pages Google hasn't crawled yet (URL is unknown to Google, Discovered - currently not indexed, or not checked yet) are sent to Google's Indexing API, newest first, shared across your sites: up to 200 a day per Google Cloud project (Google's default quota), each page at most once a week until it's indexed. Be aware: Google says the Indexing API is meant for job-posting and livestream pages; it is commonly used for any page, and Google may ignore such requests or withdraw the API's access. Your sites aren't penalized for it.

  • With a key file, turn on the Web Search Indexing API in the key's Google Cloud project (https://console.cloud.google.com/apis/library/indexing.googleapis.com?project=<project>). Until then google.push.note says so, with that link.
  • With Connect Google, the consent page also asks for Submit data to Google for indexing. Accounts connected before need to connect again to push; the note says so.
  • google.push shows pushed_today, pushed_total and indexed_after_push (pushed pages Google has indexed since), and the pages table says when each page was pushed.

Each page Google left out also links to Search Console's inspector (google_inspect_url), where you can click Request indexing by hand.

Quotas. Google allows 2,000 inspections per site per day and 600 a minute; thelastapi uses at most 200 a day per project, so big sites are covered over several days. Sync now (POST /v1/projects/:id/search/sync, vibiz search sync) runs the setup and the day's reading at once, 3 times per 10 minutes per project.

1,000 sites per Google account. That's Google's limit. Connect more than one Google account (Connect another Google account) and new sites go to the one with the fewest. Disconnecting one revokes thelastapi's access at Google; its sites stay in your Search Console, and its projects move to another connected account on the next sync.

When it runs. Each project with a Google or Bing connection is set up and read about once a day (24 hours after its last sync, plus up to an hour so many sites don't all run in the same minute); google.next_sync_at says when. After a production deploy goes live, the setup steps run again a minute later, so a fresh verification file is verified at once. A sync cut short (the service restarted mid-pass) runs again within 2 hours, not the next day. Google's per-minute limit (adding many sites at once) and a DNS record still spreading are tried again in 15 minutes, and the note says which. A call to Google, Bing or IndexNow that hangs gives up after 30 seconds and is tried again on the next run.

When something fails, google.note says what failed, why and how to fix it: no own domain yet, the verification file waiting for the next deploy, a DNS record still spreading (tried again in 15 minutes), no sitemap, or "Google access was removed; connect again" when the permission was withdrawn in the Google account (the connection shows Needs attention in Settings ▸ Search).

Bing Webmaster

Optional: IndexNow already tells Bing about new pages after every deploy. Paste your Bing Webmaster API key once (Bing Webmaster Tools ▸ Settings ▸ API access ▸ API Key) in Settings ▸ Search (POST /v1/search/bing { "api_key" }). Each site is then added to Bing Webmaster, proved with BingSiteAuth.xml (added to every production deployment once Bing gives its code), its sitemap submitted, and read daily: pages in Bing's index, pages crawled and crawl errors, clicks, impressions and top searches over 28 days (bing in GET /v1/projects/:id/search, and Bing's columns on the SEO page). When Bing gives no code, bing.note says so: in Bing Webmaster Tools, Import from Google Search Console verifies every site in one click once Google is connected here.

Pages sent to Bing directly. Once a site is verified, every sync also sends its sitemap pages Bing hasn't been sent yet through Bing Webmaster's URL submission, newest first, within the site's daily quota Bing reports (bing.submitted_today, bing.quota_left, bing.submitted_total). That's on top of IndexNow, and it works without the key file.

IndexNow retries on its own. When IndexNow refuses a send (the key file not accepted yet, too many sends), it is tried again after 1 hour, 6 hours and 24 hours, as well as after the next deploy.

The API

CallWhat it does
GET /v1/projects/:id/searchindexnow: host sent, sending_free_address, submit_free_address, key_url, pages_sent, last_sent_at, last_sent_count, note. google: connected, account_email, property, verify_method, verify_file_url, verified_at, sitemap_submitted_at, last_sync_at, next_sync_at, note, totals (pages_known, indexed, not_indexed, not_crawled (of not_indexed: pages Google has not crawled yet, waiting rather than rejected), not_inspected), clicks_28d, impressions_28d, position_28d, daily, top_queries, top_pages, push (enabled, pushed_today, pushed_total, indexed_after_push, note). bing: connected, site_added_at, verified_at, indexed, crawled_28d, crawl_errors, blocked_by_robots, indexed_daily, top_queries, submitted_today, submitted_total, quota_left, clicks_28d, impressions_28d, webmaster_url, note
PATCH /v1/projects/:id/search{ "submit_free_address": true } also sends the free address while there's no own domain
POST /v1/projects/:id/search/submitSend new and changed pages to IndexNow now; run says how many, or why none
GET /v1/projects/:id/search/pages?state=indexed|not_indexed|all&limit&offset: each sitemap page as Google sees it, with google_inspect_url
POST /v1/projects/:id/search/syncSet up and read Google and Bing now
GET /v1/search/overviewEvery site at once, most clicks first (the console's SEO ▸ All sites): host, property, verified_at, pages, clicks_28d, impressions_28d, ctr_28d, position_28d, note, bing, plus totals
POST /v1/search/google/keyOwner: { key, owner_email? }, a Google service-account key, checked with Google first
GET /v1/search/connectionsThe account's Google accounts (account_email, sites of site_limit 1,000, status, error) and Bing key. Tokens and keys are never returned
POST /v1/search/google/connectOwner: { url }, Google's consent page
POST /v1/search/bingOwner: { "api_key" }, checked with Bing first
PATCH /v1/search/settingsOwner: { "google_push": true } pushes pages Google hasn't crawled to its Indexing API (off by default)
DELETE /v1/search/connections/:idOwner: disconnect a Google account or the Bing key

The SDK has vibiz.search.get, pages, sync, connections, connectGoogle, addGoogleKey, connectBing, setGooglePush and disconnect; the vibiz_search tool has the actions get, submit, free_address, pages, sync and google_push.

Running your own Stack

Connecting Google needs a Google Cloud OAuth client (web application) with the Google Search Console API and the Site Verification API enabled, the scopes openid, email, https://www.googleapis.com/auth/webmasters, https://www.googleapis.com/auth/siteverification and (for Push to Google, optional at sign-in) https://www.googleapis.com/auth/indexing on its consent screen, the Web Search Indexing API enabled, and the redirect URI <public URL>/v1/search/google/callback. Set VIBIZ_SEARCH_GOOGLE_CLIENT_ID and VIBIZ_SEARCH_GOOGLE_CLIENT_SECRET; without them Connect Google answers not_configured and IndexNow keeps working.