Swap Quotes fails (or works inconsistently) for embedded/interpolated Ruby strings

[rusterholz]

Description

The "Swap Quotes" function, intended to switch a double-quoted string to a single-quoted string or vice versa, sometimes does nothing, but results in an error dumped to the console. (This does not always occur with every attempt to swap quotes -- sometimes it works. It is inconsistent.)

This is with Sublime Text v3.2.2, build 3211 (which states "no updates available") and BracketHighlighter v2.29.0 (believed to be the most recent version because Package Control says no updates are available).

This occurs in Ruby files (.rb suffix). I am just using the built-in Ruby support in Sublime Text 3, no additional plugins or packages. It seems to occur most-often when I attempt to swap the quotes of a string that is inside another string's interpolation. For example, in the following statement:

config.x.error_source = "#{ENV.fetch('HOSTNAME', "name-of-app")}-#{ENV.fetch('ENVIRONMENT', Rails.env)}"

attempting to swap the quotes of 'HOSTNAME', "name-of-app", or 'ENVIRONMENT' will all fail. This is true for both the single-quoted strings like 'HOSTNAME' as well as the double-quoted string "name-of-app".

Swap Quotes also sometimes fails on strings that are not part of another string's interpolation. For example, attempting to swap the outer double-quotes of the example string above for single-quotes does work on the first try (resulting in '#{ENV.fetch(\'HOSTNAME\', "name-of-app")}-#{ENV.fetch(\'ENVIRONMENT\', Rails.env)}', but attempting to then re-swap those single-quotes back to double-quotes also fails.

Support Info

• ST ver.: 3211
• Platform: osx
• Arch: x64
• Plugin ver.: 2.29.0
• Install via PC: True
• mdpopups ver.: 3.7.5
• backrefs ver.: Version could not be acquired!
• markdown ver.: 3.1.1
• pygments ver.: 2.1a0
• jinja2 ver.: 2.10.1

Steps to Reproduce Issue

1. With a ruby file open, place the cursor inside a single-quoted string, especially one that is inside another string's interpolation.
2. From the Tools menu, choose Packages --> BracketHighlighter --> Swap Quotes (or use your configured keyboard shortcut.

Expected Behavior

The single-quotes around the string should be replaced with double-quotes.

Actual Behavior

No visible change is made in the open file, but the following error sometimes appears on the Sublime Text console:

  File "/Applications/Sublime Text.app/Contents/MacOS/sublime_plugin.py", line 1082, in run_
    return self.run(edit, **args)
  File "/Users/andyrusterholz/Library/Application Support/Sublime Text 3/Installed Packages/BracketHighlighter.sublime-package/bh_core.py", line 889, in run
  File "/Users/andyrusterholz/Library/Application Support/Sublime Text 3/Installed Packages/BracketHighlighter.sublime-package/bh_core.py", line 895, in execute
  File "/Users/andyrusterholz/Library/Application Support/Sublime Text 3/Installed Packages/BracketHighlighter.sublime-package/bh_core.py", line 400, in match
  File "/Users/andyrusterholz/Library/Application Support/Sublime Text 3/Installed Packages/BracketHighlighter.sublime-package/bh_core.py", line 140, in init_match
  File "/Users/andyrusterholz/Library/Application Support/Sublime Text 3/Installed Packages/BracketHighlighter.sublime-package/bh_core.py", line 127, in refresh_rules
  File "/Users/andyrusterholz/Library/Application Support/Sublime Text 3/Installed Packages/BracketHighlighter.sublime-package/bh_rules.py", line 206, in load_rules
  File "/Users/andyrusterholz/Library/Application Support/Sublime Text 3/Installed Packages/BracketHighlighter.sublime-package/bh_rules.py", line 257, in parse_bracket_definition
  File "/Users/andyrusterholz/Library/Application Support/Sublime Text 3/Installed Packages/BracketHighlighter.sublime-package/bh_logging.py", line 19, in debug
AttributeError: 'NoneType' object has no attribute 'load_settings'```

[facelessuser]

Can you give me context? Language this is tried in (if an external package is used for the language, please specify what package). Code that you are trying it on (a minimal reproducible example).

@gir-bot add S: more-info-needed

[rusterholz]

Sure! I will also update my original description with this info:

1. This occurs in Ruby files (.rb suffix). I am just using the built-in Ruby support in Sublime Text 3, no additional plugins or packages.
2. This does not always occur with every attempt to swap quotes -- sometimes it works. It is inconsistent.
3. It seems to occur most-often when I attempt to swap the quotes of a string that is inside another string's interpolation. For example, in the following statement:

config.x.error_source = "#{ENV.fetch('HOSTNAME', "name-of-app")}-#{ENV.fetch('ENVIRONMENT', Rails.env)}"

attempting to swap the quotes of 'HOSTNAME', "name-of-app", or 'ENVIRONMENT' will all fail. This is true for both the single-quoted strings like 'HOSTNAME' as well as the double-quoted string "name-of-app".

Curiously, attempting to swap the outer double-quotes of the entire string for single-quotes does work on the first try (resulting in '#{ENV.fetch(\'HOSTNAME\', "name-of-app")}-#{ENV.fetch(\'ENVIRONMENT\', Rails.env)}', but attempting to then re-swap those single-quotes back to double-quotes also fails. This is an example of a failure that occurs on a string that is not part of another string's interpolation.

[rusterholz]

Two other things I've also noticed during my testing today:

1. Sometimes no error message appears on the console -- that is, sometimes this failure is totally silent. At any given time, though, Sublime Text appears to either be in a mode where all failure result in console messages, or no failures result in console messages -- there is never a situation where some attempts to Swap Quotes fail with messages and others fail without messages. I have not been able to determine what makes the difference.
2. The failures may occur in places that previously worked. For example, when I wrote my earlier comment, I was able to swap the outer double-quotes of the entire string to single-quotes, but not swap them back -- but right now, with no other changes made except reverting the file to its original state on disk, I can no longer swap the outer double-quotes to single-quotes like I was able to ten minutes ago, and no error message shows up in the console.

[facelessuser]

I'm not sure what the errors are, I'd have to see them here to debug such errors, but I can tell what part of the issue is, and I'm not sure I can easily offer a fix. Maybe, but let me explain.

You'll notice that the quotes around the cursor are not matching, instead the round brackets are:

This is because we are doing sub-bracket matching. There are certain brackets that are hard to match, and those are ones where the start are the same as the end.

Why is this hard? Because BH matches in a sliding window, so when there isn't a clear start syntax, and a bracket slides into that window, we aren't sure if it is a start or an end bracket.

So, how do we match them then? Well, we first look at scope. In this case, we look at the string scope. We see the cursor is on a string, so we capture the full extent of the scope (the text within that scope) and check the ends and see if they are quotes. Now strings don't normally have the same context as the rest of the code as they can be literally anything, and scope no longer comes into play as a way we can filter things out, so we then do a limited sub-bracket match inside strings, usually just round, square, and braces.

You can imagine that it can be difficult to match string scope inside string scope. And you can see here that everything is a string:

We don't have any logic to do scope bracket matches inside scope bracket matches, and I'm not sure how far the rabbit hole can go with interpolation strings.

[facelessuser]

I'll at least see if I can figure out why you are getting inconsistent results, but I'm not sure I have an easy way to fix nested scope brackets. I can maybe take a look and see if there is something clever that can be done, but BH does have some limits.

[facelessuser]

I see a couple of issues here:

1. When quote swapping was written, it from the perspective that string content is not code, so when you swap quotes, it unescapes unlike quotes and escapes like quotes. But interpolation strings can have real code inside them with real separate strings. Actual strings should not be escaped and unescaped inside an interpolation string.
2. "Swap quotes" doesn't really have logic right now to avoid escaping or unescaping strings inside strings.

We can add logic to avoid the above case because what is happening below is wrong. We swapped the outer quotes from double to single, and then single quotes inside were escaped, but 'HOSTNAME' is not really part of the parent string, it is a string within the interpolation part of the parent string.

'#{ENV.fetch(\'HOSTNAME\', "name-of-app")}-#{ENV.fetch(\'ENVIRONMENT\', Rails.env)}'

Another issue, as I stated above, we only match a limited subset of brackets within strings, strings are not matched inside other strings. I'm not sure yet how, or if, I would address this. One answer could be that we don't match strings inside of strings. I'll need time to think about this. Maybe there is some other approach we can take.

[rusterholz]

I know very little about Sublime Text scopes and other internals, and I'm not very fluent in python, so I'm just kind of spitballing here -- feel free to tell me that any portion of this is impossible!

When I look at the scope of one of the nested strings, it shows up to me as a string.quoted.double.ruby scope inside a source.ruby.embedded scope:

In fact, for multiply-nested strings (which I tried mostly as an experiment -- nobody should do this in the wild), a new source.ruby.embedded scope is created with the entire ruby expression for every level of nesting:

In your earlier comment you said:

Now strings don't normally have the same context as the rest of the code as they can be literally anything, and scope no longer comes into play as a way we can filter things out, so we then do a limited sub-bracket match inside strings, usually just round, square, and braces.

Would it be possible to do a "full" (non-limited) match inside the innermost source.ruby.embedded scope? Or, more generally, is it possible to limit the matching logic to only looking within a particular outer scope? If this was possible, then BH wouldn't have to know it was trying to match inside a string and "fall back" to the scope bracket match in this situation. The regular matching logic should work as long as it was limited to only looking at the chunk of ruby that Sublime has identified as embedded.

Am I off in left field somewhere, or is this plausible? Thanks!

[rusterholz]

Thinking more about the embedded scopes -- at the very least, perhaps whatever code BH uses to determine "am I inside a string?" could know about the existence of the embedded scopes. E.G. "If I'm in a string, but I'm also in an embedded scope inside that string, then I'm not in a string (and I can use regular matching here)."

Again, just spitballing, not sure if this is a viable option.

[facelessuser]

@rusterholz, it is definitely something to explore. The nesting stuff can get tricky though as you have to probably specify each allowed nesting configuration. Anyways, it is no small issue though, and will require a lot of thought when I have time to devote to it.

[rusterholz]

Thanks for your consideration on this so far! I'll keep an eye here for updates. 👍