Skip to content

[codex] Fix robots.txt text replacements#355

Open
imjlk wants to merge 4 commits intoSimply-Static:masterfrom
imjlk:codex/fix-robots-txt-replacement
Open

[codex] Fix robots.txt text replacements#355
imjlk wants to merge 4 commits intoSimply-Static:masterfrom
imjlk:codex/fix-robots-txt-replacement

Conversation

@imjlk
Copy link
Copy Markdown
Contributor

@imjlk imjlk commented Mar 28, 2026

Related to #159

Summary

  • centralize plain-text URL replacement for robots.txt, llms.txt, _redirects, and _headers
  • reuse that shared replacement flow across the existing SEO integrations so text-file exports follow the same URL conversion rules
  • rewrite origin URL prefixes before host-only replacement so subdirectory installs resolve exported paths correctly
  • normalize Sitemap: directives in robots.txt for absolute exports
  • process rule files during the final Local Directory transfer so an included robots.txt is not overwritten by an unprocessed copy

Root cause

Plain-text file replacement was implemented outside the main URL conversion flow and handled in multiple places.

In typical WordPress setups, SEO plugins often add a Sitemap: directive to robots.txt using the live WordPress origin URL and a plugin-specific sitemap filename, such as sitemap_index.xml or sitemap.xml. That is correct on the live site, but when exporting with absolute destination URLs, the final static robots.txt should point to the static destination URL instead of the origin host.

This created two gaps:

  • text-file replacement only handled some origin-based values consistently, which could leave incorrect paths behind in cases like subdirectory installs
  • even when robots.txt was processed earlier in the export pipeline, the final Local Directory transfer could copy the original rule file again and overwrite the already-processed result

With this change, text-file replacements go through shared utilities, Sitemap: rewriting is limited to absolute exports, and final rule-file writes preserve the processed content.

Validation

  • php -l src/class-ss-util.php
  • php -l src/handlers/class-ss-text-file-handler.php
  • php -l src/handlers/class-ss-rule-file-handler.php
  • php -l src/integrations/class-ss-aio-seo-integration.php
  • php -l src/integrations/class-ss-rank-math-integration.php
  • php -l src/integrations/class-ss-yoast-integration.php

@imjlk imjlk marked this pull request as ready for review March 28, 2026 11:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant