The necessity for static analysis of source code …
It highly likely most Java (and not only) developers have used at minimum some sort of a static analysis tool to perform a task such as (to name a few):
discovering bugs, vulnerabilities or code smells such as unused variables (what popular IDEs typically do);
performing automated refactoring or code completion;
enforcing code and quality standards.
To perform static code analysis we typically need a proper representation of source code, suitable for analysis. A programming language can be described by a formal grammar. Further more a parser can be created or generated following the rules of a formal grammar to create proper representation (typically a parse tree) from source code. Based on the type of language we want to represent we can use different types of formal grammars:
regular grammars (i.e. regular expressions): they are available in most programming languages but are used typically for more basic parsing tasks as in many cases it is not suitable (or possible) to parse a modern programming langugage, it is slow and it is hard to maintain the grammar (i.e. the regular expression);
context-free grammars (i.e. BNF or eBNF): one of the most well-known formats is BNF (and its variants), a parser can also be generated by the grammar rules;
other formal grammars (i.e. PEG).
It is not uncommon that in the early days different tools for static code analysis required writing a parser manually which is not a trivial task ….
Parser generators to the rescue …
Tools can be created to generate parsers based on a target context-free grammar rules. This is, for example, the case with tools like LEX and YACC written in C and generating code in C. At a high level parser generation is illustrated by the following diagram:
In the early days of Java Sun Microsystems has developed a parser generator called Jack which was then later renamed to JavaCC (which stands for Java compiler-compiler). Another popular alternative for generating a parser for a Java grammar is ANTLR (ANother Tool for Language Recognition). Both of these parser generators are well supported and written in Java. JavaCC (similar to YACC for C) can combine grammar rules with Java code that is included in the generated parser, however JavaCC provides code generation only for Java while ANTLR is general purpose, has a large number of grammars for a number of programming languages and provides the possibility to generate parsers in different languages. Both of these tools work with formal grammars in eBNF form and considering for example the above general diagram here’s how the process of parser generation looks like in Antlr with a simple example of an expression parser generator:
The parse tree that is generated by the parser itself provides more effort in terms of code analysis so that is why typically parser generators provide the possibility to generate a more concise representation which eliminates extra symbols and provides additional symbol resolution capabilities: the AST (abstract syntax tree). The process of using ANTLR or JavaCC in a standard Maven/Gradle project is very similar.
For ANTLR:
create grammar files under src/main/antlr4 (in g4 format, Java 20 grammar files available here)
add Antlr4 dependency and plugin in Maven/Gradle build file
generate lexer and parser using Maven/Gradle build
Once this is in place the generated parser can be used to generate a parse tree that can have i.e. a listener attach for specific executions during the parsing process. Example using Antlr-generated parser:
String content = "public class Example { public void func(int x){ return x + 10; } }";
Java20Lexer lexer = new Java20Lexer(CharStreams.fromString(content));
CommonTokenStream tokens = new CommonTokenStream(lexer);
Java20Parser parser = new Java20Parser(tokens);
ParseTree tree = parser.compilationUnit();
ParseTreeWalker walker = new ParseTreeWalker();
ExprListener listener = new ExprListener();
walker.walk(listener, tree);
An alternative way to create a parser is by a parsing expression grammar (PEG). A library that implements this approach (the grammar rules are written in Java code directly as part of the application) is Parboiled.
Java libraries to the rescue …
Parser generators and PEG parsers are quite generic. They also may not be up to date with the desired Java version. As an alternative a specializing parsing library can be used such as JavaParser or Eclipse JDT.
JavaParser
It is based on JavaCC, it is well maintained and provides support for JDK 21. It provides enhanced symbol resolution and generates and AST from the source code. In addition it provides capabilities to query the AST via a DSL provided by the library, generate code from the AST or modify it. It is really simple to get started using the library, the following example counts the number of methods in a class:
public static int countMethods(File file) throws FileNotFoundException {
CompilationUnit cu = StaticJavaParser.parse(file);
int count = 0;
for (Node node : cu.findAll(MethodDeclaration.class)) {
count++;
}
return count;
}
To get started using JavaParser it is sufficient to include the following dependency:
Eclipse JDT (Java Developer Tools) is the main fuel behind the Java editor in Eclipse IDE that provides advanced capabilities like partial compilation, code completion etc. In earlier days of Eclipse it was not straighforward to use JDT outside of the Eclipse IDE primarily because these requred a number of extra dependencies to be dragged as well. Now Eclipse JDT is available as standalone library via the following dependency:
