# Search

> Every production deploy sends the site's new and changed pages to IndexNow, and with the account's own Google Search Console connected each site is verified, its sitemap submitted and its indexed pages and searches read every day.

Search gets a business's pages in front of search engines without anyone remembering to do it. Console ▸ **SEO** shows every site at once (pick a domain for its full view), and each project has the same view under ▸ **SEO**, `vibiz search`, the `vibiz_search` tool or `GET /v1/projects/:id/search`.

## After every deploy: IndexNow

When a production deploy goes live (a build, a promote or a rollback), thelastapi reads the site's `/sitemap.xml` (every child sitemap of a sitemap index too) and sends the pages that are **new, or whose `<lastmod>` changed**, to [IndexNow](https://www.indexnow.org). IndexNow hands them to Bing, Yandex, Naver and Seznam, usually within minutes. A site with no sitemap sends its home page once.

- **Nothing to set up.** IndexNow proves the site is yours with a key file at `/<key>.txt`. Every production deployment gets it, in `public/` for apps with a `package.json` (or `static/` when the app already has one), at the root for a plain static site, or inside the one folder a built site is served from (like `site/`, holding its `index.html` and its sitemap or `robots.txt`) when there's no homepage at the root. Google's and Bing's proof files go to the same place. An app that ships its own file at that path keeps it.
- **Own domain only.** Pages are sent for the project's first live domain. The free `<slug>.vibiz.app` address is not sent unless the owner turns on **Also send the free vibiz.app address** (`vibiz search free-address on`): half-built projects shouldn't get indexed under the shared domain, and a business that moves to its own domain later starts clean. Once a domain is live, only the domain is sent.
- **Only what changed.** Each page is sent once, then again when its `lastmod` changes. A new domain starts the list over.
- **Send now** (`vibiz search submit`, `POST /v1/projects/:id/search/submit`) sends without a deploy.

When a run sends nothing, `note` says why: no own domain yet, or the key file isn't served yet (the first deploy after Search arrived adds it; deploy again).

Google doesn't take IndexNow: it reads the sitemap, and the account's own Google Search Console (below) is how it hears about the site.

## Google Search Console

Add a **Google key** once, in console ▸ **Settings ▸ Search** (or `vibiz search connect google --key key.json --owner-email you@gmail.com`, or `POST /v1/search/google/key` with `{ key, owner_email? }`). Nobody signs in with Google:

1. In [Google Cloud](https://console.cloud.google.com), pick or create a project and turn on the **Search Console API** and the **Site Verification API**.
2. **IAM ▸ Service accounts ▸ Create service account** (any name, no roles), then **Keys ▸ Add key ▸ Create new key ▸ JSON**. Paste or choose that file.
3. Optionally give your own Google email: it is added as an owner of every site, so they also appear in your own Search Console.

The key is checked with Google before it is kept (sealed, never shown again), and an API that is off is named with the link to turn it on. A key holds up to 1,000 sites; add another one when it fills up. Only the account owner adds keys.

Or sign in with Google instead (**Or sign in with Google**, `POST /v1/search/google/connect`), when the Stack has a Google sign-in client set up.

From then on, every project with a live own domain (or the free address, when it is sent) is set up in **your** Search Console, with no step of yours:

1. **Proved yours.** A domain bought through thelastapi is proved with a DNS record we add to its zone, and becomes a Domain property (`sc-domain:example.com`, covering `www` and every subdomain). Any other domain, and the free address, is proved with Google's verification file (`/google<code>.html`), added to every production deployment the same way as IndexNow's key; it is verified a minute after the next production deploy goes live, and the file stays on later deploys, as Google checks again now and then.
2. **Added** to Search Console as a property of the connected Google account.
3. **Sitemap submitted**: `https://<domain>/sitemap.xml`, once the site serves one.
4. **Every day**: clicks, impressions and average position by day, the top 50 searches and the top 50 pages over 28 days; and up to **200 pages a day** inspected (URL Inspection): pages never checked first, then pages whose `lastmod` changed since, then the longest ago. Each gets Google's verdict and words (`Submitted and indexed`, `Crawled - currently not indexed`, `Discovered - currently not indexed`…), the last crawl, and the canonical Google chose.

The project's **Search** tab (and `GET /v1/projects/:id/search`, `vibiz search`) shows indexed and not-indexed pages, clicks, impressions and average position, a 28-day chart, the pages table and the top searches and pages. Mission Control's board shows `42 · 31 idx` in the Pages cell (sitemap pages · indexed) and a **Search clicks 7d** column.

**Push to Google (opt-in, off by default).** Turn on **Push pages to Google** in Settings ▸ Search (`PATCH /v1/search/settings { "google_push": true }`, `vibiz search push on`) and every day the pages Google hasn't crawled yet (*URL is unknown to Google*, *Discovered - currently not indexed*, or not checked yet) are sent to Google's **Indexing API**, newest first, shared across your sites: up to **200 a day per Google Cloud project** (Google's default quota), each page at most once a week until it's indexed. Be aware: Google says the Indexing API is meant for job-posting and livestream pages; it is commonly used for any page, and Google may ignore such requests or withdraw the API's access. Your sites aren't penalized for it.

- **With a key file**, turn on the **Web Search Indexing API** in the key's Google Cloud project (`https://console.cloud.google.com/apis/library/indexing.googleapis.com?project=<project>`). Until then `google.push.note` says so, with that link.
- **With Connect Google**, the consent page also asks for *Submit data to Google for indexing*. Accounts connected before need to connect again to push; the note says so.
- `google.push` shows `pushed_today`, `pushed_total` and `indexed_after_push` (pushed pages Google has indexed since), and the pages table says when each page was pushed.

Each page Google left out also links to Search Console's inspector (`google_inspect_url`), where you can click **Request indexing** by hand.

**Quotas.** Google allows 2,000 inspections per site per day and 600 a minute; thelastapi uses at most 200 a day per project, so big sites are covered over several days. **Sync now** (`POST /v1/projects/:id/search/sync`, `vibiz search sync`) runs the setup and the day's reading at once, 3 times per 10 minutes per project.

**1,000 sites per Google account.** That's Google's limit. Connect more than one Google account (Connect another Google account) and new sites go to the one with the fewest. Disconnecting one revokes thelastapi's access at Google; its sites stay in your Search Console, and its projects move to another connected account on the next sync.

**When it runs.** Each project with a Google or Bing connection is set up and read about once a day (24 hours after its last sync, plus up to an hour so many sites don't all run in the same minute); `google.next_sync_at` says when. After a production deploy goes live, the setup steps run again a minute later, so a fresh verification file is verified at once. A sync cut short (the service restarted mid-pass) runs again within 2 hours, not the next day. Google's per-minute limit (adding many sites at once) and a DNS record still spreading are tried again in 15 minutes, and the note says which. A call to Google, Bing or IndexNow that hangs gives up after 30 seconds and is tried again on the next run.

When something fails, `google.note` says what failed, why and how to fix it: no own domain yet, the verification file waiting for the next deploy, a DNS record still spreading (tried again in 15 minutes), no sitemap, or **"Google access was removed; connect again"** when the permission was withdrawn in the Google account (the connection shows *Needs attention* in Settings ▸ Search).

## Bing Webmaster

Optional: IndexNow already tells Bing about new pages after every deploy. Paste your Bing Webmaster API key once (Bing Webmaster Tools ▸ Settings ▸ API access ▸ API Key) in **Settings ▸ Search** (`POST /v1/search/bing { "api_key" }`). Each site is then added to Bing Webmaster, proved with `BingSiteAuth.xml` (added to every production deployment once Bing gives its code), its sitemap submitted, and read daily: pages in Bing's index, pages crawled and crawl errors, clicks, impressions and top searches over 28 days (`bing` in `GET /v1/projects/:id/search`, and Bing's columns on the SEO page). When Bing gives no code, `bing.note` says so: in Bing Webmaster Tools, **Import from Google Search Console** verifies every site in one click once Google is connected here.

**Pages sent to Bing directly.** Once a site is verified, every sync also sends its sitemap pages Bing hasn't been sent yet through Bing Webmaster's URL submission, newest first, within the site's daily quota Bing reports (`bing.submitted_today`, `bing.quota_left`, `bing.submitted_total`). That's on top of IndexNow, and it works without the key file.

**IndexNow retries on its own.** When IndexNow refuses a send (the key file not accepted yet, too many sends), it is tried again after 1 hour, 6 hours and 24 hours, as well as after the next deploy.

## The API

| Call | What it does |
|---|---|
| `GET /v1/projects/:id/search` | `indexnow`: `host` sent, `sending_free_address`, `submit_free_address`, `key_url`, `pages_sent`, `last_sent_at`, `last_sent_count`, `note`. `google`: `connected`, `account_email`, `property`, `verify_method`, `verify_file_url`, `verified_at`, `sitemap_submitted_at`, `last_sync_at`, `next_sync_at`, `note`, `totals` (`pages_known`, `indexed`, `not_indexed`, `not_crawled` (of not_indexed: pages Google has not crawled yet, waiting rather than rejected), `not_inspected`), `clicks_28d`, `impressions_28d`, `position_28d`, `daily`, `top_queries`, `top_pages`, `push` (`enabled`, `pushed_today`, `pushed_total`, `indexed_after_push`, `note`). `bing`: `connected`, `site_added_at`, `verified_at`, `indexed`, `crawled_28d`, `crawl_errors`, `blocked_by_robots`, `indexed_daily`, `top_queries`, `submitted_today`, `submitted_total`, `quota_left`, `clicks_28d`, `impressions_28d`, `webmaster_url`, `note` |
| `PATCH /v1/projects/:id/search` | `{ "submit_free_address": true }` also sends the free address while there's no own domain |
| `POST /v1/projects/:id/search/submit` | Send new and changed pages to IndexNow now; `run` says how many, or why none |
| `GET /v1/projects/:id/search/pages` | `?state=indexed\|not_indexed\|all&limit&offset`: each sitemap page as Google sees it, with `google_inspect_url` |
| `POST /v1/projects/:id/search/sync` | Set up and read Google and Bing now |
| `GET /v1/search/overview` | Every site at once, most clicks first (the console's **SEO ▸ All sites**): `host`, `property`, `verified_at`, `pages`, `clicks_28d`, `impressions_28d`, `ctr_28d`, `position_28d`, `note`, `bing`, plus `totals` |
| `POST /v1/search/google/key` | Owner: `{ key, owner_email? }`, a Google service-account key, checked with Google first |
| `GET /v1/search/connections` | The account's Google accounts (`account_email`, `sites` of `site_limit` 1,000, `status`, `error`) and Bing key. Tokens and keys are never returned |
| `POST /v1/search/google/connect` | Owner: `{ url }`, Google's consent page |
| `POST /v1/search/bing` | Owner: `{ "api_key" }`, checked with Bing first |
| `PATCH /v1/search/settings` | Owner: `{ "google_push": true }` pushes pages Google hasn't crawled to its Indexing API (off by default) |
| `DELETE /v1/search/connections/:id` | Owner: disconnect a Google account or the Bing key |

The SDK has `vibiz.search.get`, `pages`, `sync`, `connections`, `connectGoogle`, `addGoogleKey`, `connectBing`, `setGooglePush` and `disconnect`; the `vibiz_search` tool has the actions `get`, `submit`, `free_address`, `pages`, `sync` and `google_push`.

## Running your own Stack

Connecting Google needs a Google Cloud OAuth client (web application) with the **Google Search Console API** and the **Site Verification API** enabled, the scopes `openid`, `email`, `https://www.googleapis.com/auth/webmasters`, `https://www.googleapis.com/auth/siteverification` and (for Push to Google, optional at sign-in) `https://www.googleapis.com/auth/indexing` on its consent screen, the **Web Search Indexing API** enabled, and the redirect URI `<public URL>/v1/search/google/callback`. Set `VIBIZ_SEARCH_GOOGLE_CLIENT_ID` and `VIBIZ_SEARCH_GOOGLE_CLIENT_SECRET`; without them Connect Google answers `not_configured` and IndexNow keeps working.
