How to generate a parser for a custom format in Go#

Your Go service reads a config format with nested blocks, and its hand-written parser grows a special case with every bug report. Each Go parser generator you try asks for something different: a grammar file, struct tags, or a Java build step. The first PEG grammar you write can also double its work for every level the blocks nest.

What you get

You will end up with one config format parsed three ways in Go and a test that holds the three to one result. A count catches the grammar whose work doubles per level of nesting. This is for you if a Go service owns its format.

Short answer

Write the grammar as a PEG file, let pigeon generate one Go source file from a //go:generate directive, and commit the output, which needs only the standard library. participle skips generation by reading the grammar from struct tags, and the ANTLR4 Go target needs Java and a runtime module. pigeon caches no results by default, so restructure a backtracking rule or pass Memoize(true), and count the work with its Statistics option.

You will need

Go 1.24 or later, and Java only to regenerate the ANTLR parser. Verified 2026-10-21 against Go 1.24.7, pigeon 1.3.0, participle 2.1.4, the ANTLR 4.13.2 tool with its Go runtime 4.13.1, and OpenJDK 21.0.12.1. The sample parses one service config three ways, compares the results, and counts the work two pigeon grammars do on nested input. Reporting several errors from one parse is covered in recover from syntax errors and report several in one parse.

Approaches compared

pigeon and ANTLR both keep the grammar in a file a reviewer can read. pigeon’s output imports only the standard library, while the ANTLR output needs its runtime module, plus a Java toolchain whenever the grammar changes. participle has no generation step, so changing the format means changing a struct, and the grammar exists only as tags until you print it.

Write the grammar as a PEG file

pigeon reads a PEG grammar whose actions are blocks of Go code and writes one Go source file, as its package documentation describes. The actions build the shared model types, so the parser returns the value the program wants.

Config <- _ es:(Entry _)* EOF {
	return entries(es), nil
}

Entry <- Block / Attr

// A block may end with a semicolon. Writing that as `(_ ';')?` keeps the
// parse linear; see ../pegnaive for the version that does not.
Block <- name:Ident _ label:Label? _ '{' _ es:(Entry _)* '}' (_ ';')? {
	b := &model.Block{Name: name.(string), Entries: entries(es)}
	if label != nil {
		b.Label = label.(string)
	}
	return b, nil
}

Attr <- key:Ident _ '=' _ value:Value {
	return &model.Attr{Key: key.(string), Value: value}, nil
}

A //go:generate directive records the exact command beside the grammar. The go command never runs it during a build, so the generated file is committed and regenerated by hand.¹

// The generated parser is committed. Regenerate it after editing config.peg.
//go:generate go run github.com/mna/pigeon@v1.3.0 -o config.go config.peg

go generate ./pegconfig && wc -l pegconfig/config.go

1869 pegconfig/config.go

Those 1869 lines import only the standard library and the model package, so pigeon adds nothing to go.mod. The (_ ';')? at the end of Block makes the trailing semicolon optional inside one rule, which matters more than it looks.

Put the grammar in struct tags

participle builds its parser from the types it fills. Each field’s tag is a fragment of the grammar, and MustBuild assembles them when the program starts. Use the parser:"..." form of the tag, because the bare form is not a valid Go struct tag and go vet rejects it, a clash the tag syntax notes warn about. Give an integer field a type of its own, as decimal does here, because participle fills an int64 with strconv.ParseInt in base 0, which reads 010 as octal 8.

type config struct {
	Entries []*entry `parser:"@@*"`
}

type entry struct {
	Block *block `parser:"  @@"`
	Attr  *attr  `parser:"| @@"`
}

type block struct {
	Name    string   `parser:"@Ident"`
	Label   *string  `parser:"@String?"`
	Entries []*entry `parser:"'{' @@* '}' ';'?"`
}

type attr struct {
	Key   string `parser:"@Ident '='"`
	Value value  `parser:"@@"`
}

type value struct {
	String *string  `parser:"  @String"`
	Int    *decimal `parser:"| @Int"`
	Bool   *string  `parser:"| @('true' | 'false')"`
}

go run ./cmd/ebnf

Config = Entry* .
Entry = Block | Attr .
Block = <ident> <string>? "{" Entry* "}" ";"? .
Attr = <ident> "=" Value .
Value = <string> | <int> | ("true" | "false") .

The EBNF that participle prints from the tags is the nearest this approach comes to a grammar file, and it reads like the PEG rules in the first step. Nothing is generated, so a change to the format is a change to a struct.

Generate the parser with ANTLR

ANTLR generates a lexer and a parser from a .g4 grammar for any of its ten target languages, Go among them with -Dlanguage=Go. The Go output depends on the github.com/antlr4-go/antlr/v4 runtime module. The tool itself is a Java program, and the Go target’s guide suggests committing its jar to your repository.

grammar Config;

config : entry* EOF ;
entry  : block | attr ;
block  : name STRING? '{' entry* '}' ';'? ;
attr   : name '=' value ;
// true and false lex as BOOL, so a name takes BOOL as well as IDENT.
name   : IDENT | BOOL ;
value  : STRING | INT | BOOL ;

BOOL    : 'true' | 'false' ;
IDENT   : [a-zA-Z_] [a-zA-Z0-9_]* ;
STRING  : '"' ~["]* '"' ;
INT     : '-'? [0-9]+ ;
COMMENT : '#' ~[\n]* -> skip ;
WS      : [ \t\r\n]+ -> skip ;

The directive in antlrconfig/build.go runs java -jar ../tools/antlr-4.13.2-complete.jar -Dlanguage=Go -package antlrconfig -no-listener Config.g4, which ran by hand to produce the committed files.

wc -l antlrconfig/config_lexer.go antlrconfig/config_parser.go && go vet ./antlrconfig

  143 antlrconfig/config_lexer.go
  927 antlrconfig/config_parser.go
 1070 total
# example.com/configparse/antlrconfig
antlrconfig/config_parser.go:270:2: unreachable code
antlrconfig/config_parser.go:398:2: unreachable code
antlrconfig/config_parser.go:612:2: unreachable code
antlrconfig/config_parser.go:733:2: unreachable code
antlrconfig/config_parser.go:827:2: unreachable code
antlrconfig/config_parser.go:926:2: unreachable code

The tool wrote two files with 1070 lines, and go vet objects to six of them, one per rule. Each generated rule ends with goto errorExit after its return, a trick its own comment says prevents an unused-label error, and vet reports the line as unreachable. Keep the generated package out of your vet run, or accept the noise.

Compare the three on one file

main.go parses service.conf with all three, compares the results through one canonical printout, and then hands each parser two broken files.

go run .

pigeon      built the config below
participle  same
antlr       same

service "api" {
  port = 8080
  debug = false
  tls {
    cert = "/etc/api/cert.pem"
    key = "/etc/api/key.pem"
  }
  route "health" {
    path = "/healthz"
    timeout_ms = 250
  }
}

"service {\n  port 8080\n}\n"
  pigeon      bad.conf:2:8 (17): no match found, expected: "#", "=", "\"", "{" or [ \t\r\n]
  participle  bad.conf:2:3: unexpected token "port" (expected "}" ";"?)
  antlr       2:8: no viable alternative at input 'port8080'

"service {\n  port = 8080\n"
  pigeon      bad.conf:3:1 (24): no match found, expected: "#", "}", [ \t\r\n] or [a-zA-Z_]
  participle  bad.conf:3:1: unexpected token "<EOF>" (expected "}" ";"?)
  antlr       3:1: extraneous input '<EOF>' expecting {'}', BOOL, IDENT}

All three build the same config. Their errors differ in where they point: pigeon and ANTLR name the token after port, where the = belongs, while participle names port, the start of the entry it could not finish. ANTLR counts columns from zero, so build.go adds one before reporting. It also hands over a string token as written, quotes and backslashes included, so build.go reads it with strconv.Unquote, as pigeon’s action does. The module file shows what each approach adds at run time.

require (
	github.com/alecthomas/participle/v2 v2.1.4
	github.com/antlr4-go/antlr/v4 v4.13.1
)

require golang.org/x/exp v0.0.0-20240506185415-9bf2ced13842 // indirect

Check it worked

parse_test.go holds the three parsers to one result on service.conf and on 19 inputs where their lexers could part ways, such as escapes, leading zeros, and true as a key. It also checks that each refuses both broken files, and pins the counts measured in the next section.

go test -v ./... | grep -E '^(=== RUN|PASS$|FAIL$)'

=== RUN   TestAllThreeBuildTheSameConfig
=== RUN   TestAllThreeAgreeOnTrickyInput
=== RUN   TestEveryParserRefusesABrokenFile
=== RUN   TestTheNaiveGrammarDoublesItsWorkPerLevel
=== RUN   TestMemoizeAndTheRestructuredGrammarStayLinear
=== RUN   TestMaxExpressionsStopsARunawayParse
PASS

Six tests pass. TestTheNaiveGrammarDoublesItsWorkPerLevel fails if a pigeon release starts to cache results by default, which is a change worth hearing about.

When it goes wrong

The work doubles with every level of nesting. pegnaive/config.peg spells the optional semicolon as two alternatives, the way a format’s prose description tends to list them.

// A block may end with a semicolon, spelled here as two alternatives, the
// way the format's description lists them. When the first fails at the end,
// the second parses the whole block again, every nested block included.
Block <- name:Ident _ label:Label? _ '{' _ es:(Entry _)* '}' _ ';' {
	return block(name, label, es), nil
} / name:Ident _ label:Label? _ '{' _ es:(Entry _)* '}' {
	return block(name, label, es), nil
}

pigeon caches no results unless asked, and its Statistics option counts the expressions a parse evaluates. A count, unlike a timing, comes out the same on every machine.

go run ./cmd/memo

nesting         naive    naive+Memoize   restructured
1                 540              239            252
2                1248              352            365
4                5496              578            591
8               90456             1030           1043
12            1462104             1486           1499
16           23457624             1946           1959

naive, 16 levels, MaxExpressions(100000): nested.conf:19:1 (158): rule Block: max number of expressions parsed

Restructure the rule, as pegconfig does, and the count stays linear with no cache. Pass Memoize(true) when you cannot restructure, and benchmark it on ordinary files too: the generated documentation warns that it costs memory and time on typical input.² For input from someone else, MaxExpressions turns a runaway parse into an error that names the rule.

A broken file parses without an error. The default ANTLR error listener prints to stderr, recovers, and returns a tree, with no error value for the caller to check.

go run ./cmd/partial 2>&1

line 2:7 no viable alternative at input 'port8080'
returned a tree with 1 entry: service{port8080}<EOF>

Remove the default listener and collect the errors, as build.go does, before you trust any tree.

When not to do this

Do not write a grammar for a format that HCL, TOML, or YAML already covers. The config here is a near relative of HCL, whose Go package parses blocks, labels, and attributes with no grammar for you to maintain.

Do not choose ANTLR for a format only Go reads. Its tooling and its other targets pay off when another language needs the same parser; without one, you carry Java, a jar, and a runtime module for nothing.

Do not let a pigeon parser read untrusted input without MaxExpressions, even once its grammar is restructured. A rule that backtracks is a slow request waiting for the right file. Restructuring stops the backtracking but not the recursion: 200,000 levels of a{, a 600 KB file, end the restructured parser with a fatal stack overflow, which recover cannot catch.

Do not hand the format to participle if people outside the team must review its grammar, since what they would review is struct tags.

Related how-tos

• Remove left recursion from a PEG grammar

• Recover from syntax errors and report several in one parse

• Write a recursive descent parser by hand in TypeScript

• Parse one config format identically in TypeScript and Go

• Write a Pratt parser for operator precedence in filters

Last verified

Verified 2026-10-21 against Go 1.24.7, pigeon 1.3.0, participle 2.1.4, and the ANTLR Go runtime 4.13.1. Every output block is what the command preceding it printed. The ANTLR 4.13.2 tool ran by hand on OpenJDK 21.0.12.1 to generate antlrconfig, from a jar checked against the SHA-1 that Maven Central publishes; the checked commands use the committed output. No timing was measured. The counts are expression evaluations, so they repeat on any machine. The fatal stack overflow was run by hand, since it needs over a gigabyte of memory; the tests check only that MaxExpressions stops the same nesting.