Verilog/VHDL parser

[Nic30]

There are some parsers for HDL however all of them have some ridiculous weakness.

I would like to use hdlConvertor because I know that the Python dependency is not tied with the parsing code and potential missing features can be implemented and there are ANTLR4 grammars that can be used as a map to code.

However netlistDB compatibility will cost 100+ M/H of work for sure.

[Nic30]

@Thomasb81 I am thinking about major update of hdlConvertor. I would like to implement full process parsing FIRRTL, separate python/c++ part and finalize preprocessor and comment parsing.

Do you know about some better c++ compatible library for parsing of HDL languages?
If I start implementing this features will you be able to check if it is breaking something for you?
I can fix the problems in our systems, however I do not have much contact on other peoples which are using hdlConvertor.

[Thomasb81]

Hi Nic,

FIRRTL is unknown to me. This will be an additional parser that would live concurrently to verilog and vhdl ?
What could be the impact on existing code ? I don't see for me any usage of this description language (Because not seen in the industry for the moment).

I would as a first step limit the FIRRTL work to the interface as it is yet more or less for verilog and vhdl. But I don't know your requirement and motivation regarding FIRRTL...

Actually I still work on verilog/systemVerilog preprocessor to be able to process UVM code, and then systemVerilog support, I think sv grammar is in a good shape but I really need an sv preprocessor to pickup code from anywhere on the net.

I don't use the tool yet, so you won't break anything, except if you consider to completely revisit the json structure...

Thomas

[Nic30]

FIRRTL is unknown to me. This will be an additional parser that would live concurrently to verilog and vhdl ?

Yes it is much more simple HDL than Verilog/VHDL, it is used by risc-v community. (Maybe you heard about chise3, the FIRRTL is used in it.) I also do not need it I just think that it would be nice to have compatibility with the chisel3 based tools.

I don't use the tool yet, so you won't break anything, except if you consider to completely revisit the json structure...

OK, I will see how much time will remain and what can be done.

[Thomasb81]

Actually I still work on verilog/systemVerilog preprocessor to be able to process UVM code, and then systemVerilog support, I think sv grammar is in a good shape but I really need an sv preprocessor to pickup code from anywhere on the net.

Actually the code is there : https://github.com/Thomasb81/hdlConvertor/tree/sv
But not yet ready to be merged.
Some directive like FILE and LINE need to be fix to return correct result.
Also I fear to have performance issue : The preprocessing of the UVM library take more than 5-6 minutes when the library is compiled in debug mode and ~1min26 in release mode. This mean that each time you have following directive :
include "uvm_macros.svh"
The preprocessing take 2 to 6 minutes per file depending of the optimization... it seems too long... Even if I don't know exactly what to expect.

[Nic30]

Do you know exactly what operation is slow?
Maybe we could save "precompiled AST" for later use.
Or we can build python objects from ANTLR4 AST on demand.
Or maybe do you have debug messages enabled? Maybe debug messages are making this too slow?

[Thomasb81]

I did not yet investigate too much on what is slow.
But on my testcase I have no print at all except the result text on stdout.

$python setup.py build --build-type Debug -- -- -j 16
[...]
$time python tests/test_thomas.py
[...]
real	5m20,704s
user	5m20,605s
sys	0m0,032s
$python setup.py build -- -- -j 16
[...]
$time python tests/test_thomas.py
[...]
real	1m19,352s
user	1m19,301s
sys	0m0,036s

The test(tests/test_thomas.py not available in not publish) is about to read about 17 files (through `include directive) representing a little bit more than 312Ko of text source code.
1m19 with compiler optimization for processing 17 files / 312ko seems quite long to me.

I am not expert on antlr4.
But the ast is the representation of the source code in memory ? right ? what would be the benefice to save this if I read the source code only one time ?

I saw a response from @mike-lischke on antlr4 mailing list saying that "left hand predicate, in all lexer rules" is performance killer. I use left hand predicate in the parser...

I need to do additional test...

[Thomasb81]

Ok I got my issue.
I recursively call my preprocessor function (build the ast then run the parser each time) instead of visit node ...
On ifdef elseif else / ifndef elseif else which are big part of the case that popup the issue I improve the performance by 2x.
Thomas

[Thomasb81]

Do you know exactly what operation is slow?
Maybe we could save "precompiled AST" for later use.
Or we can build python objects from ANTLR4 AST on demand.
Or maybe do you have debug messages enabled? Maybe debug messages are making this too slow?

It seems the slow part is the tree build :
tree::ParseTree *tree = parser->file();
file is the rule entry point of the preprocessor...

I am wondering if it is not the NEWLINE token that I mention everywhere in grammar rules and only required to be not ignored for `define end rule...

[Nic30]

@Thomasb81

That usually means that the grammar is backtracking a lot.
I do not have time now however this month I plane to do significant improvements in hdlConvertor.