cue/parser: allow omitting commas in lists

CUE already allows struct field commas to be
omitted at end-of-line via automatic comma
insertion (scanner sets insertEOL=true after
`]`, identifiers, literals, etc.). Lists
required explicit commas between elements.

This removes that restriction for language
version v0.17.0 and later: a newline after a
list element now acts as a comma, consistent
with struct fields.

The parser's synthetic-comma guard in
parseListElement (which raised "missing ','
before newline in list literal") is retained
for files with a language version below
v0.17.0, preserving backwards compatibility.

A new versionAtLeast helper reads the file
version via p.experiments.LanguageVersion()
and compares with semver, matching the
pattern already used by compile.go.

The spec's exception note ("the parser will
require explicit commas between two list
elements") and its TODO comment are removed;
a versioned note is kept in an HTML comment.

Fixes a latent bug where [expr\nfor x in xs
{z}] silently dropped the comprehension (the
old guard returned ok=false when it saw FOR
after the synthetic comma, breaking the parse
loop before the comprehension was reached).

Signed-off-by: Marcel van Lohuizen <mpvl@gmail.com>
Change-Id: If6ed8764449547482a1f9848099c5ef9f30a16b9
Reviewed-on: https://cue.gerrithub.io/c/cue-lang/cue/+/1235981
Reviewed-by: Roger Peppe <rogpeppe@gmail.com>
Unity-Result: CUE porcuepine <cue.porcuepine@gmail.com>
TryBot-Result: CUEcueckoo <cueckoo@cuelang.org>

Marcel van Lohuizen 3 weeks ago 70a9648f 1b32beb8

+66 -13

3 changed files

expand all

cue

parser

parser.go

parser_test.go

doc

ref

spec.md

+20 -7

cue/parser/parser.go

··· 27 27 "cuelang.org/go/cue/token" 28 28 "cuelang.org/go/internal" 29 29 "cuelang.org/go/internal/cueexperiment" 30 + "cuelang.org/go/internal/mod/semver" 30 31 ) 31 32 32 33 // The parser structure holds the parser's internal state. ··· 92 93 p.comments = &commentState{pos: -1} 93 94 94 95 p.next() 96 + } 97 + 98 + // versionAtLeast reports whether the file's language version is at least v. 99 + // An absent version (empty string) is treated as the latest version. 100 + func (p *parser) versionAtLeast(v string) bool { 101 + fv := "" 102 + if p.experiments != nil { 103 + fv = p.experiments.LanguageVersion() 104 + } 105 + return fv == "" || semver.Compare(fv, v) >= 0 95 106 } 96 107 97 108 type commentState struct { ··· 1440 1451 expr = p.parseBinaryExprTail(token.LowestPrec+1, expr) 1441 1452 expr = p.parseAlias(expr) 1442 1453 1443 - // Enforce there is an explicit comma. We could also allow the 1444 - // omission of commas in lists, but this gives rise to some ambiguities 1445 - // with list comprehensions. 1446 - if p.tok == token.COMMA && p.lit != "," { 1454 + // Before v0.17.0 the parser required explicit commas between list 1455 + // elements; automatic (newline-inserted) commas were not accepted. 1456 + if !p.versionAtLeast("v0.17.0") && p.tok == token.COMMA && p.lit != "," { 1447 1457 p.next() 1448 - // Allow missing comma for last element, though, to be compliant 1449 - // with JSON. 1458 + // Allow missing comma for last element to be compliant with JSON. 1450 1459 if p.tok == token.RBRACK || p.tok == token.FOR || p.tok == token.IF { 1451 1460 return expr, false 1452 1461 } 1453 1462 p.errf(p.pos, "missing ',' before newline in list literal") 1454 - } else if !p.atComma("list literal", token.RBRACK, token.FOR, token.IF) { 1463 + goto exit 1464 + } 1465 + 1466 + if !p.atComma("list literal", token.RBRACK, token.FOR, token.IF) { 1455 1467 return expr, false 1456 1468 } 1469 + exit: 1457 1470 p.next() 1458 1471 1459 1472 return expr, true

+41

cue/parser/parser_test.go

··· 503 503 out: `{a: [1, 2, 3, b, c, ...], b: [1, 2, 3], c: [1, 2, 3], d: [1+2, 2, 4]}`, 504 504 }, 505 505 { 506 + desc: "lists with optional commas", 507 + version: "v0.17.0", 508 + in: `{ 509 + a: [ 510 + 1 511 + 2 512 + 3 513 + ] 514 + b: [ 515 + "foo" 516 + "bar" 517 + ] 518 + c: [1 519 + 2] 520 + }`, 521 + out: `{a: [1, 2, 3], b: ["foo", "bar"], c: [1, 2]}`, 522 + }, 523 + { 524 + desc: "list element followed by comprehension on next line", 525 + version: "v0.17.0", 526 + in: `{ 527 + a: [1, 2, 3] 528 + b: [ 529 + 0 530 + for x in a { x } 531 + ] 532 + }`, 533 + out: `{a: [1, 2, 3], b: [0, for x in a {x}]}`, 534 + }, 535 + { 536 + desc: "lists with optional commas not allowed before v0.17.0", 537 + version: "v0.16.0", 538 + in: `{ 539 + a: [ 540 + 1 541 + 2 542 + ] 543 + }`, 544 + out: "{a: [1, <*ast.BadExpr>]}\nmissing ',' before newline in list literal (and 2 more errors)", 545 + }, 546 + { 506 547 desc: "list types", 507 548 in: `{ 508 549 a: 4*[int]

+5 -6

doc/ref/spec.md

··· 193 193 - an ellipsis `...` 194 194 195 195 196 - Although commas are automatically inserted, the parser will require 197 - explicit commas between two list elements. 196 + To reflect idiomatic use, examples in this document elide commas using 197 + these rules. 198 198 199 199  202 - 203 - To reflect idiomatic use, examples in this document elide commas using 204 - these rules. 205 204 206 205 207 206 ### Identifiers

Configure Feed

Configure Feed