[swift][cpd] Exception when running for Swift 4 code (KeyPath)

[nalexn]

cpd stopped working after I migrated the source code to Swift 4.

The problem: pmd cpd quits with an exception.

A full command showcasing the problem:
pmd cpd --language swift --minimum-tokens 50 --files source.swift

Result of running the command above:

net.sourceforge.pmd.lang.ast.TokenMgrError: Lexical error in file <path to file> at line 62, column 60.  Encountered: token recognition error at: '\.'
	at net.sourceforge.pmd.cpd.SwiftTokenizer.tokenize(SwiftTokenizer.java:48)
	at net.sourceforge.pmd.cpd.CPD.addAndThrowLexicalError(CPD.java:142)
	at net.sourceforge.pmd.cpd.CPD.add(CPD.java:137)
	at net.sourceforge.pmd.cpd.CPD.add(CPD.java:107)
	at net.sourceforge.pmd.cpd.CPD.add(CPD.java:69)
	at net.sourceforge.pmd.cpd.CPD.addDirectory(CPD.java:79)
	at net.sourceforge.pmd.cpd.CPD.addRecursively(CPD.java:64)
	at net.sourceforge.pmd.cpd.CPDCommandLineInterface.addSourcesFilesToCPD(CPDCommandLineInterface.java:125)
	at net.sourceforge.pmd.cpd.CPDCommandLineInterface.addSourceFilesToCPD(CPDCommandLineInterface.java:105)
	at net.sourceforge.pmd.cpd.CPDCommandLineInterface.main(CPDCommandLineInterface.java:83)
	at net.sourceforge.pmd.cpd.CPD.main(CPD.java:177)

The line of code for which cpd throws an exception is:

let observation = scrollView.observe(\.contentOffset, options: .new) { [weak self] (scrollView, _) in

Aparently cpd cannot handle the new Swift 4 syntax for KeyPath, which is \.

[jsotuyod]

@nalexn you are right. The current Swift grammar is heavily outdated, being based on Swift 2.x (I'm not sure if it was 2.2 or 2.3).

We have a pending issue to upgrade the grammar, however, we are not certain on how to deal with the old versions... Unfortunately, none of the PMD maintainers are too familiar with it to pinpoint which parts of the grammar where introduced (or removed / changed) at each version, so doing multi-version support as we have on Java is currently very hard for us...

On the other hand, migrating to 4 and dropping support for older versions may hinder legacy projects.

We are looking for people willing to help us out on this, by both providing feedback on the course of action we should take; and dealing with the multi-version support (the grammar itself can already be obtained from Antlr, no need to manually write it, just add the proper version checks / version compatibility).

[kenji21]

As Swift.g4 comes from sleekbytes/tailor, and they have added "#keyPath" support:

sleekbyte/tailor@ac71cff#diff-afab8183c2d9a2a8dc197c667c432d26

I've patched to also support Swift4 KeyPaths: sleekbyte/tailor#493

[kenji21]

Here is my commit : openium@1ab96d4

I can send a PR if necessary

[jsotuyod]

@kenji21 You sir are amazing!

We would love to get a PR with you for proper acknowledgement of your efforts.

The commit you show looks great for a PR, the one thing I'd ask you to add however is to change the test swift file being parsed to use both the old and the new keyPath grammar somewhere, so that we can assure that it's being properly parsed without errors.

[jsotuyod]

If my understanding is correct, that PR would rise our Swift support to Swift 3 (3.2?), and have only support for the new keypath of Swift 4.

Full swift 4 support would still be missing:

• keypath shorthand notation supported by @kenji21
• Multiline strings
• One-Sided Ranges (I believe the current grammar is lax enough to support it, in cases such as planets[..<4] but it would have to be tested; other uses such as pattern patching are not)
• Generic Subscripts (I believe these are supported by the current lax grammar, but would have to be tested)
• Associated Type Constraints
• Class and Protocol Existential

Source: https://www.raywenderlich.com/163857/whats-new-swift-4

[kenji21]

Okay, as the grammar file is "exported" from sleekbyte/tailor, do having only tests in their project seems right to you @jsotuyod ?

[kenji21]

to keep swift3 support, maybe having a "--language swift3" support using the old "grammar file" can be achievable (but I don't know the pmd source code enough to do that).

[jsotuyod]

@kenji21 as far as I know, the grammar is incremental, so all Swift 3 is valid Swift 4 code. That means we can eventually have a scheme similar to what we have in Java with a single grammar file. Don't worry about it for the time being.

As for tests, just using the new syntax somewhere in BTree.swift should be enough.

[kenji21]

Ok, committed for multiline strings, and added one-sided ranges (their were already managed by the current grammar) : openium@b10e21d

[kenji21]

Associated Type Constraints and Class and Protocol Existential are also already managed (the where keyword and & operator are already in the grammar) openium@56cead8

Maybe I can send a PR now ?

[jsotuyod]

@kenji21 whenever you see fit. I'll try to figure out how to build the version checks for Antlr grammars, but shouldn't be too different from JavaCC's.

[kenji21]

PR updated for swift 4.1