@atomist/antlr

Integration with ANTLR for TypeScript for Atomist automation clients.

ANTLR is a powerful parser generator, for which many grammars are available.

This enables running path expressions against ANTLR ASTs in a consistent manner to ASTs produced by other grammars.

Includes Java support, in the JavaFileParser implementation of FileParser, as an example, test and for actual Java support, as well as Kotlin support, to show how to handle a grammar with a distinct lexer and parser. Other ANTLR grammars can be integrated with Atomist using this project as library, including offering an update model producing clean diffs.

See overall path expression documentation.

Usage

First, create an instance of a FileParser or FileParserRegistry.

A FileParser knows how to parse files using a single grammar: for example, MicrogrammarBasedFileParser (from automation client) that uses a single microgrammar, or JavaFileParser from this project. A FileParserRegistry can accommodate multiple FileParser instances, determining whichever is appropriate to execute a given path expression.

Then use the findMatches method in astUtils, which has the following signature:

export function findMatches(p: ProjectNonBlocking,
                            globPattern: string,
                            parserOrRegistry: FileParser | FileParserRegistry,
                            pathExpression: string | PathExpression): Promise<TreeNode[]> {

The following example looks in all Java files in a project for a given path expression:

findMatches(project, JavaFiles, JavaFileParser,
    "//variableDeclaratorId/Identifier")
    .then(matches => {
        ...

Returned matches are updatable after project flushing.

SPI: Supporting other ANTLR grammars

This project includes support for parsing Java using the Java ANTLR grammar.

There are many available ANTLR grammars, and the same approach can be used with most of them, making it possible to work with their ASTs in a consistent manner with Atomist.

To add support for another grammar, perform the following steps:

Generate TypeScript

Use the ANTLR CLI to generate the necessary files from the grammar. It is installed as a development dependency of this project.

If your grammar has a separate lexer, first generate the TypeScript for that. In this example, we'll use the Kotlin grammar.

$ ./node_modules/.bin/antlr4ts -visitor lib/tree/ast/antlr/kotlin/KotlinLexer.g4

Then generate the TypeScript for the parser.

$ ./node_modules/.bin/antlr4ts -lib lib/tree/ast/antlr/kotlin -visitor lib/tree/ast/antlr/kotlin/KotlinParser.g4

You may need to reorder some of the generated code to eliminate forward references to ensure compilation. In your antlr-gen/XXXXParser file, reorder to put all classes before the XXXXParser class.

If you are using tslint you may need to disable it for the generated sources as in this project.

Note that antlr4ts is written in Java, and npm only wraps it, so you'll need a recent JVM. A JVM is not needed to execute the resulting grammar, which is pure TypeScript. Documentation for the ANTLR CLI is here.

Create a FileParser Implementation to Integrate your Grammar with your Atomist Automation Client

Create an instance of FileParser to work with your grammar. Use TreeBuildingListener to do the work of building an Atomist TreeNode from the ANTLR parse tree.

Create your FileParser in the style of JavaFileParser in this project, passing the top level production name and the lexer and parser classes.

const JavaFileParser = new AntlrFileParser("compilationUnit", JavaLexer, JavaParser);

If your grammar uses Java or other code you may need to port that code to JavaScript or TypeScript. Please refer to ANTLR documentation in this case. Please validate your grammar with an IDE plugin or other tool before generating code and attempting integration.