Skip to content

Break pseudo token#422

Merged
leejet merged 3 commits into
leejet:masterfrom
daniandtheweb:break_wip
Nov 16, 2025
Merged

Break pseudo token#422
leejet merged 3 commits into
leejet:masterfrom
daniandtheweb:break_wip

Conversation

@daniandtheweb

@daniandtheweb daniandtheweb commented Sep 27, 2024

Copy link
Copy Markdown
Contributor

I've noticed that #285 hasn't been updated for a while so I've upstreamed the PR and fixed the regex to correctly identify the BREAK token without deleting the white spaces.
@ursg I've still marked you as the author as most of the work was yours, if you still want to update your PR instead I'll close this.

@daniandtheweb daniandtheweb force-pushed the break_wip branch 4 times, most recently from a966b4a to 3124fa0 Compare September 28, 2024 14:36
@lostdisc

Copy link
Copy Markdown

So THAT'S how you deal with color bleed! Good to know, and do want!

@wbruna

wbruna commented Jun 19, 2025

Copy link
Copy Markdown
Contributor

That 'B' stuff on the regex took me a bit too long to understand, so a comment would've been appreciated 🙂 Apart from that, the code looks good to me, and worked as expected on a few tests.

@daniandtheweb

daniandtheweb commented Jun 19, 2025

Copy link
Copy Markdown
Contributor Author

Actually I should refactor this code to properly adapt it to the master branch. Right now it's a bit messy. I'll still add the original author as coauthor.

@daniandtheweb

Copy link
Copy Markdown
Contributor Author

I've finally remembered that I had this PR pending. I've just upstreamed it and it seems to work nicely.
@wbruna I added a comment on what does B do for easier understanding, let me know if it seems clear enough, I'm not really good at explaining regex.

Comment thread util.cpp
// (abc:3.12) - increases attention to abc by a multiplier of 3.12
// [abc] - decreases attention to abc by a multiplier of 1.1
// BREAK - separates the prompt into conceptually distinct parts for sequential processing
// B - internal helper pattern; prevents 'B' in 'BREAK' from being consumed as normal text

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is enough to make the regex a little less magical. Thanks!

daniandtheweb and others added 2 commits November 10, 2025 05:10
Co-authored-by: Urs Ganse <urs.ganse@helsinki.fi>
@leejet leejet merged commit 6448430 into leejet:master Nov 16, 2025
9 checks passed
@leejet

leejet commented Nov 16, 2025

Copy link
Copy Markdown
Owner

Thank you for your contribution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants