Lexical error when parsing Integer values

[JanPeter]

I'm currently struggling with a strange problem. I want to parse the following snipped:

  type User {
    name: String = "Ignatz"
    loggedIn: Boolean = false
  }

  type Task {
    done: Boolean = true
    priority: Number = 54
  }

When I create my grammar and try to parse this snipped, I get the following error message:

[Error: Lexical error on line 9. Unrecognized text.
    ... priority: Number = 54      }
    -----------------------^]

The funny thing is, if I paste my grammar into a tool like this and choose a LRLA(1) grammar as parser type i get no errors. That's the way I create my grammar on my local machine:

./node_modules/.bin/jison --file src/grammar.json -p lalr -o src/grammar.js

That's my grammar as json:

{
  "lex": {
    "macros": {
      "digit": "[0-9]",
      "esc": "\\\\",
      "int": "-?([0-9]|[1-9][0-9]+)",
      "exp": "(?:[eE][-+]?[0-9]+)",
      "frac": "(?:\\.[0-9]+)"
    },
    "rules": [
      ["\\s+", "/* skip whitespace */"],
      ["type", "return 'TYPE'"],
      ["implements", "return 'IMPLEMENTS'"],
      ["{int}{frac}?{exp}?\\b", "return 'NUMBER';"],
      ["\"(?:{esc}[\"bfnrt/{esc}]|{esc}u[a-fA-F0-9]{4}|[^\"{esc}])*\"", "yytext = yytext.substr(1,yyleng-2); return 'STRING';"],
      ["(false|true)", "return 'BOOLEAN'"],
      ["(String|Number|Boolean)", "return 'SCALAR'"],
      ["[a-zA-Z]+", "return 'WORD'"],
      ["\\{", "return '{'"],
      ["\\}", "return '}'"],
      ["\\[", "return '['"],
      ["\\]", "return ']'"],
      ["\\(", "return '('"],
      ["\\)", "return ')'"],
      [":", "return ':'"],
      [",", "return ','"],
      [";", "return ';'"],
      ["!", "return '!'"],
      ["=", "return '='"],
      ["#", "return '#'"],
      ["->", "return '->'"],
      ["<-", "return '<-'"]
    ]
  },

  "bnf": {
    "Schema": [["ModelList", "return $1;"]],
    "ModelList": [["Model", "$$ = [ $1 ];"],
                  ["ModelList Model", "$$ = $1; $1.push($2)"]],
    "Model": [["ModelHeader ModelProperties", "$$ = $1; $1.properties = $2;"],
              ["ModelHeader ModelProperties ;", "$$ = $1; $1.properties = $2;"]],
    "ModelHeader": [["ModelName", "$$ = { name: $1 };"],
                    ["ModelName IMPLEMENTS WORD", "$$ = { name: $1, implements: $3 };"]],
    "ModelName": [["TYPE WORD", "$$ = $2;"]],
    "ModelProperties": [["{ }", "$$ = [];"],
                        ["{ PropertyList }", "$$ = $2;"]],
    "PropertyList": [["Property", "$$ = []; $$.push($1);"],
                     ["PropertyList Property", "$$ = $1; $1.push($2);"]],
    "Property": [[ "WORD : PropertyType", "$$ = $3; $3.name = $1;" ],
                 [ "WORD : PropertyType ,", "$$ = $3; $3.name = $1;" ],
                 [ "WORD : PropertyType ;", "$$ = $3; $3.name = $1;" ]],
    "Value": [["STRING", "$$ = $1;"],
              ["NUMBER", "$$ = Number(yytext);"],
              ["BOOLEAN", "$$ = $1;"]],
    "PropertyTypeScalar": [["SCALAR", "$$ = { type: $1, scalar: true };"],
                           ["SCALAR !", "$$ = { type: $1, scalar: true, required: true };"]],
    "PropertyTypeScalarValue": [[ "PropertyTypeScalar = Value", "$$ = $1; $1.value = $3" ]],
    "PropertyType": [["PropertyTypeScalar", "$$ = $1"],
                     ["PropertyTypeScalarValue", "$$ = $1"],
                     ["WORD PropertyRelation", "$$ = $2; $2.model = $1;"],
                     ["[ WORD ] PropertyRelation", "$$ = $4; $4.model = $2; $4.many = true;"]],
    "PropertyRelation": [[ "# <-", "$$ = { type: 'Relation', direction: 'from' };" ],
                         [ "# ->", "$$ = { type: 'Relation', direction: 'to' };" ],
                         [ "# <- ( WORD )", "$$ = { label: $4, type: 'Relation', direction: 'from' };" ],
                         [ "# -> ( WORD )", "$$ = { label: $4, type: 'Relation', direction: 'to' };" ]]
  }
}

I'm using the latest jison version 0.4.18

[GerHobbelt]

Hm, odd. This does work with my own jison 0.6.0-191 (npm: jison-gho).

You might want to go and have a look at the internals, see what jison generated as code for your lexer, in particular the table with the regexes for your lexer rules. (The lexer is always the bottom half of the generated js parser/lexer source code file; the lexer tables are near the bottom of the file.)

[JanPeter]

I ended up switching to nearleyjs which works fine for me. So a contributor can decide if he wants to close this issue, or try to fix it.

Just curious, why did you chose to fork this library instead of contributing to it?

[GerHobbelt]

I did contribute as well, but had specific needs for my own production environment when swift turn-around / speed of development was a requirement. Managing a slew of single-subject/independent pullreqs (extracting them in such a way as to make them independent, setting up and then tracking each pullreq) next to the CTO day job wasnt a realistic choice re time costs, so it started with a few pullreqs, got complex pretty darn fast and now you have #338. I'ld love to hear from @zaach if/when/how he wants to merge (some of?) the work done some day, but its not just trivial changes under the hood and clearly his needs and time constraints and mine re jison were (are?) quite different at the time when i worked/work on jison a lot due to our/my needs for it at the day job. So i settled for the next best option: a fork. I dont like it, but its the best option available under the circumstances. I do a lot of work in bursts, which does not mix well with the meditate-on-a-mountain management model that (by necessity) applies to a lot of OSS as its volunteer driven rather than business driven. I like OSS a lot, prefer it above all else, but the way of cooperating where people with very diverse needs and availability work on a project is still a tough nut I haven't cracked to my satisfaction. (note that i say "work" rather than "contribute": this is about how to process change at a rate and depth beyond the trivial) Given my own limitations, I've developed a personal process where I can track & merge others' work pretty quickly (sans the need for pullreq) when my time allows spending attention on the given project, so I settled for that, despite the drawbacks, until a merge is viable. If you like what has been done, you might consider having a look at how to mix this into the mainline. That's not my talent and it's much appreciated if you can apply yours to it. #338 is there for that reason: I dont want a fork, mine just happens to be one.

…

On 13 Sep 2017 9:45 am, "Jan Peter" ***@***.***> wrote: I ended up switching to nearleyjs which works fine for me. So a contributor can decide if he wants to close this issue, or try to fix it. Just curious, why did you chose to fork this library instead of contributing to it? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#360 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAYkHnM1wRVJPnH5GHR-7PNwIUTvXB_Yks5sh4gFgaJpZM4PPcP3> .

[GerHobbelt]

Thanks for the https://github.com/Hardmath123/nearley reference, by the way.

[JanPeter]

Well, thanks for that detailed explanation! I would've also taken that route 👍
Wish I knew about your fork before rewriting my grammar to this nearley syntax...

[metasansana]

Wow, thanks for this discussion guys. I have been wondering about your fork for a while @GerHobbelt. Just installed it and trying it out in a few existing projects.