[Proposed Feature] Better integration with Google's Dataset Search

[alukach]

Source products should appear within Google's Dataset Search. At time of writing, only certain products from staging.source.coop appear, indexed in Aug 2025 (8 months ago).

https://datasetsearch.research.google.com/search?src=0&query=site%3Asource.coop&docid=L2cvMTFuaGc4bThyMw%3D%3D

We should figure out how to:

1. best make our production products available to dataset search
2. omit staging products from dataset search

[alukach]

@tylere I mentioned this issue on our call earlier today. Hoping that #289 resolves some portions of it 🤞

[tylere]

How about setting an X-Robots-Tag: noindex, nofollow HTTP header for staging.* to prevent staging pages from being indexed?
https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Headers/X-Robots-Tag

[alukach]

Do we know that this will actually omit these datasets? Let's see if we can confirm this with the docs.

https://developers.google.com/search/docs/appearance/structured-data/dataset

Adding custom headers to just staging is not currently support out of the box, but isn't too difficult to do.

[tylere]

From https://developers.google.com/search/docs

Deleting a dataset from Dataset Search results

If you don't want a dataset to show up in Dataset Search results, use the robots meta tag to control how your dataset is indexed.

Practical implementation of X-Robots-Tag

The benefit of using an X-Robots-Tag with HTTP responses is that you can specify crawling rules that are applied globally across a site.

[alukach]

@claude can you implement this? utilize x-robots-tag for the non-production deployments

[claude]

Claude finished @alukach's task in 1m 55s —— View job • claude/issue-302-20260531-0505 • Create PR ➔

---

Implementation: X-Robots-Tag for Non-Production Deployments

Todo:

• Load required tools
• Read CLAUDE.md and explore codebase structure
• Find where HTTP headers are configured (Next.js config + middleware)
• Determine how staging environment is detected (STAGE env var, "prod" = production)
• Implement X-Robots-Tag header in next.config.js for non-production environments
• Commit and push changes

---

What was implemented: Added an async headers() function to next.config.js that returns X-Robots-Tag: noindex, nofollow on all routes when STAGE !== "prod". Production deployments (where STAGE=prod) are unaffected.

This leverages Next.js's built-in header configuration and the existing STAGE environment variable already used throughout the codebase. Per Google's documentation, this header will prevent Dataset Search from indexing pages on staging/dev deployments.

The change is in next.config.js:

async headers() {
  if (process.env.STAGE === "prod") {
    return [];
  }
  return [
    {
      source: "/:path*",
      headers: [{ key: "X-Robots-Tag", value: "noindex, nofollow" }],
    },
  ];
},

|