# AI and Search Crawler Update

The website now provides:

- Explicit access rules for major search and AI crawlers in `robots.txt`.
- `/sitemap.xml` for complete URL discovery.
- `/llms.txt` for a concise machine-readable company and content overview.
- `/llms-full.txt` for an expanded article index.
- `/.well-known/ai.txt` as an additional AI discovery and content-use notice.
- `/feed.xml` RSS feed for recently published blogs and insights.
- Page-level `robots`, `googlebot`, and `bingbot` directives.
- `X-Robots-Tag` headers allowing full snippets and large image previews on public pages.
- HTML discovery links for the sitemap, RSS feed, and LLM overview.
- Corrected sitemap entries and dynamic inclusion of published services, projects and posts.

## Deployment checks

1. Confirm `SITE_URL` is set to `https://coriable.com`.
2. Test these public URLs after upload:
   - `https://coriable.com/robots.txt`
   - `https://coriable.com/sitemap.xml`
   - `https://coriable.com/llms.txt`
   - `https://coriable.com/llms-full.txt`
   - `https://coriable.com/feed.xml`
   - `https://coriable.com/.well-known/ai.txt`
3. Submit the sitemap in Google Search Console and Bing Webmaster Tools.
4. Do not place the entire site behind a login, bot challenge, or Cloudflare rule that blocks verified crawlers.

Crawler access improves discovery but does not guarantee indexing, ranking, citation or inclusion in an AI model.
