Addition: In-flight pattern mining

[tarunKoyalwar]

Summary

This PR adds an in-flight pattern mining implementation to AlterX, enabling automatic pattern discovery from subdomain datasets. The implementation uses hierarchical clustering algorithms to identify common patterns and generate DSL templates automatically.

Key Features

• Hierarchical Ngram-Based Clustering: Multi-level clustering approach combining ngram prefix matching, token extraction, and edit distance clustering
• Levenshtein Distance Clustering: Groups similar subdomains based on edit distance thresholds
• Modular Architecture: Clean separation of concerns across multiple files:
  • clustering.go: Core clustering algorithms and orchestration
  • tokenization.go: Token extraction and parsing logic
  • pattern_generation.go: DSL pattern generation pipeline
  • pm.go: Main PatternMiner interface and execution flow

Implementation Approach

This PR focuses on high-level logic and architecture with placeholder functions marked with TODOs for future implementation. The structure is designed to be:

• Well-documented with clear algorithm descriptions and examples
• Easy to test and extend
• Ready for incremental implementation

Attribution

This implementation is based on the regulator project by @cramppet. Regulator is a subdomain pattern mining tool that uses hierarchical clustering algorithms to automatically discover patterns in subdomain datasets. We've adapted and extended these concepts to provide automatic pattern generation capabilities within AlterX.

Special thanks to @cramppet for the excellent work on subdomain pattern analysis.

Testing

Test structure is in place in pm_test.go and utils_test.go for validation once the placeholder functions are implemented.

Next Steps

• Implement tokenization logic
• Implement pattern generation algorithm
• Add comprehensive test cases
• Performance optimization
• Integration with existing AlterX features

---

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch pattern-mining

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[ehsandeep]

@coderabbitai create and link github issue for this!

[coderabbitai]

Tip

For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Get PR details and comments to understand what issue needs to be created
gh pr view 281 --repo projectdiscovery/alterx --json title,body,comments --jq '.'

Length of output: 4745

---

🏁 Script executed:

#!/bin/bash
# Create a GitHub issue for tracking the implementation of TODOs in PR #281
gh issue create \
  --repo projectdiscovery/alterx \
  --title "Implement Pattern Mining TODOs" \
  --body "## Overview

This issue tracks the implementation of placeholder functions and TODOs for the in-flight pattern mining feature introduced in PR #281.

## Background

PR #281 adds the architectural foundation for pattern mining in AlterX, with a modular design across multiple files (\`clustering.go\`, \`tokenization.go\`, \`pattern_generation.go\`, \`pm.go\`). The PR focuses on high-level logic with placeholder functions marked for future implementation.

## Tasks

- [ ] Implement tokenization logic
- [ ] Implement pattern generation algorithm
- [ ] Add comprehensive test cases
- [ ] Performance optimization
- [ ] Integration with existing AlterX features

## References

- PR: https://github.com/projectdiscovery/alterx/pull/281
- Comment: https://github.com/projectdiscovery/alterx/pull/281#issuecomment-3497288412
- Requested by: @ehsandeep

## Attribution

Based on the [regulator](https://github.com/cramppet/regulator) project by [@cramppet](https://github.com/cramppet)." \
  --assignee ehsandeep

Length of output: 53

---

@ehsandeep: I've created issue #282 to track the implementation of the pattern mining TODOs from this PR.

The issue includes:

• All 5 tasks from the "Next Steps" section
• References to this PR and your comment
• Attribution to the regulator project
• Assigned to you

You can view it here: #282

[tarunKoyalwar]

This PR has been superseded by PR #283 which includes:

• Complete pattern mining implementation
• All linting errors resolved (0 issues)
• All tests passing
• Proper error handling following Go best practices
• Clean commit history

PR #283 can be safely merged. This PR can be closed.