mvdan.cc / cue
this repo has no description
0
fork

Configure Feed

Select the types of activity you want to include in your feed.

internal/core/adt: allow any CUE value in json/yaml.Validate

By default, when calling builtin functions, an error is returned
for each argument that is:

- non-concrete (i.e. builtin(`string`))
- an unresolved disjunction (i.e. builtin("a" | "b"))

In addition, all default values are resolved, so that `builtin(1 | *2)`
will be called as `builtin(2)`, losing important data in some cases
where the disjunction could be resolved to `1`.

Some examples in which this behavior is erroneous:

- `yaml.Validate("a", "a" | "b")` (results in an "unresolved
disjunction" error)
- `yaml.Validate("a", string)` (results in a non-concrete value error)
- `yaml.Validate("a", *int | string)` (even if non-concrete values
were allowed, this would result in false because the default value
`int` would be resolved in place of the disjunction, and 'a' is
not an int)

To fix this, we delegate the responsibility to evaluate disjunctions
and resolve default values to the builtin. This way each builtin can
decide whether its arguments should be concrete, and whether
disjunctions/defaults should be resolved.

Note: these are further original comments from Noam. The corresponding
code has been hoisted to a separate CL
A new CallCtxt.Schema method is added which, as opposed to
CallCtxt.Value, does not resolve arguments to their default values /
enforce concreteness.
All stdlib function registrations, located in pkg.go files, are
auto-generated based on the public functions in the corresponding
stdlib packages. Because of this, a new `internal.pkg.Schema` type
is added, so that stdlib functions that need to use CallCtxt.Schema,
can declare a parameter astype pkg.Schema.

This fixes incorrect failures when passing non-concrete values and
unresolved disjunctions to `yaml.Validate` and `json.Validate`.

A test is added that only works when the new evaluator is enabled. This
is because the old evaluator evaluated the following CUE as incomplete:

{a: 1} & ({a!: int} | {b!: int})

Which is incorect because the required fields should "force" the
expression to evaluate to `{a: 1}`.

Note that this does not yet fix cases like:

yaml.Validate("a: 1", close({a: int}) | close({b: int}))

This will require further investigation to see why the "closedness" of
the disjuncts is seemingly lost during the evaluation, resulting in a
non-concrete unification of the yaml with the disjunction - and a false
result.

Updates #2741.

Signed-off-by: Noam Dolovich <noam.tzvi.dolovich@gmail.com>
Change-Id: Id34b5ada4704df0e43ff9ea33ad47d631af9ba74
Reviewed-on: https://review.gerrithub.io/c/cue-lang/cue/+/1194425
Unity-Result: CUE porcuepine <cue.porcuepine@gmail.com>
TryBot-Result: CUEcueckoo <cueckoo@cuelang.org>
Reviewed-by: Daniel Martí <mvdan@mvdan.cc>

authored by

Noam Dolovich and committed by
Marcel van Lohuizen 1b01c835 2a843385

+146 -46
+2 -1
encoding/yaml/yaml.go
··· 23 23 "cuelang.org/go/cue" 24 24 "cuelang.org/go/cue/ast" 25 25 cueyaml "cuelang.org/go/internal/encoding/yaml" 26 + "cuelang.org/go/internal/pkg" 26 27 "cuelang.org/go/internal/source" 27 28 pkgyaml "cuelang.org/go/pkg/encoding/yaml" 28 29 ) ··· 109 110 // Validate validates the YAML and confirms it matches the constraints 110 111 // specified by v. For YAML streams, all values must match v. 111 112 func Validate(b []byte, v cue.Value) error { 112 - _, err := pkgyaml.Validate(b, v) 113 + _, err := pkgyaml.Validate(b, pkg.Schema(v)) 113 114 return err 114 115 }
+1 -1
internal/core/compile/builtin.go
··· 81 81 Name: "close", 82 82 Params: []adt.Param{structParam}, 83 83 Result: adt.StructKind, 84 - // Noncrete: true, // TODO: should probably be noncrete 84 + // NonConcrete: true, // TODO: should probably be noncrete 85 85 Func: func(c *adt.OpContext, args []adt.Value) adt.Expr { 86 86 s, ok := args[0].(*adt.Vertex) 87 87 if !ok {
+2
internal/pkg/types.go
··· 21 21 22 22 // A Schema represents an arbitrary cue.Value that can hold non-concrete values. 23 23 // By default function arguments are checked to be concrete. 24 + // 25 + // TODO(mvdan,mpvl): consider using type Schema = cue.Value. 24 26 type Schema cue.Value 25 27 26 28 func (s Schema) Value() cue.Value {
+3 -2
pkg/encoding/json/manual.go
··· 28 28 "cuelang.org/go/cue/token" 29 29 cuejson "cuelang.org/go/encoding/json" 30 30 internaljson "cuelang.org/go/internal/encoding/json" 31 + "cuelang.org/go/internal/pkg" 31 32 ) 32 33 33 34 // Compact generates the JSON-encoded src with insignificant space characters ··· 130 131 131 132 // Validate validates JSON and confirms it matches the constraints 132 133 // specified by v. 133 - func Validate(b []byte, v cue.Value) (bool, error) { 134 - err := cuejson.Validate(b, v) 134 + func Validate(b []byte, v pkg.Schema) (bool, error) { 135 + err := cuejson.Validate(b, v.Value()) 135 136 if err != nil { 136 137 return false, err 137 138 }
+3 -2
pkg/encoding/json/pkg.go
··· 118 118 {Kind: adt.BytesKind | adt.StringKind}, 119 119 {Kind: adt.TopKind}, 120 120 }, 121 - Result: adt.BoolKind, 121 + Result: adt.BoolKind, 122 + NonConcrete: true, 122 123 Func: func(c *pkg.CallCtxt) { 123 - b, v := c.Bytes(0), c.Value(1) 124 + b, v := c.Bytes(0), c.Schema(1) 124 125 if c.Do() { 125 126 c.Ret, c.Err = Validate(b, v) 126 127 }
+4 -4
pkg/encoding/json/testdata/gen.txtar
··· 125 125 } | { 126 126 b!: int 127 127 } 128 - result: json.Validate(str, schema) 128 + result: true 129 129 } 130 130 disjunctionClosed: { 131 131 str: *"{\"a\":10}" | string ··· 134 134 } | { 135 135 b: int 136 136 } 137 - result: json.Validate(str, schema) 137 + result: true 138 138 } 139 139 140 140 // Issue #2395 ··· 406 406 } | { 407 407 b!: int 408 408 } 409 - result: json.Validate(str, schema) 409 + result: true 410 410 } 411 411 disjunctionClosed: { 412 412 str: *"{\"a\":10}" | string ··· 415 415 } | { 416 416 b: int 417 417 } 418 - result: json.Validate(str, schema) 418 + result: true 419 419 } 420 420 421 421 // Issue #2395
+6 -4
pkg/encoding/yaml/manual.go
··· 84 84 return ast.NewList(a...), nil 85 85 } 86 86 87 - // Validate validates YAML and confirms it is an instance of the schema 88 - // specified by v. If the YAML source is a stream, every object must match v. 89 - func Validate(b []byte, v cue.Value) (bool, error) { 87 + // Validate validates YAML and confirms it is an instance of schema. 88 + // If the YAML source is a stream, every object must match v. 89 + func Validate(b []byte, schema pkg.Schema) (bool, error) { 90 90 d := cueyaml.NewDecoder("yaml.Validate", b) 91 + v := schema.Value() 91 92 r := v.Context() 92 93 for { 93 94 expr, err := d.Decode() ··· 132 133 // specified by v using unification. This means that b must be consistent with, 133 134 // but does not have to be an instance of v. If the YAML source is a stream, 134 135 // every object must match v. 135 - func ValidatePartial(b []byte, v cue.Value) (bool, error) { 136 + func ValidatePartial(b []byte, schema pkg.Schema) (bool, error) { 136 137 d := cueyaml.NewDecoder("yaml.ValidatePartial", b) 138 + v := schema.Value() 137 139 r := v.Context() 138 140 for { 139 141 expr, err := d.Decode()
+8 -6
pkg/encoding/yaml/pkg.go
··· 68 68 {Kind: adt.BytesKind | adt.StringKind}, 69 69 {Kind: adt.TopKind}, 70 70 }, 71 - Result: adt.BoolKind, 71 + Result: adt.BoolKind, 72 + NonConcrete: true, 72 73 Func: func(c *pkg.CallCtxt) { 73 - b, v := c.Bytes(0), c.Value(1) 74 + b, schema := c.Bytes(0), c.Schema(1) 74 75 if c.Do() { 75 - c.Ret, c.Err = Validate(b, v) 76 + c.Ret, c.Err = Validate(b, schema) 76 77 } 77 78 }, 78 79 }, { ··· 81 82 {Kind: adt.BytesKind | adt.StringKind}, 82 83 {Kind: adt.TopKind}, 83 84 }, 84 - Result: adt.BoolKind, 85 + Result: adt.BoolKind, 86 + NonConcrete: true, 85 87 Func: func(c *pkg.CallCtxt) { 86 - b, v := c.Bytes(0), c.Value(1) 88 + b, schema := c.Bytes(0), c.Schema(1) 87 89 if c.Do() { 88 - c.Ret, c.Err = ValidatePartial(b, v) 90 + c.Ret, c.Err = ValidatePartial(b, schema) 89 91 } 90 92 }, 91 93 }},
+117 -26
pkg/encoding/yaml/testdata/validate.txtar
··· 21 21 22 22 validate: #test & { 23 23 fn: yaml.Validate 24 + 25 + // TODO: fix this test: the second disjunct should be eliminated, so there 26 + // should not be a concreteness error. 27 + t1: _ 24 28 } 25 29 26 30 validatePartial: #test & { 27 31 fn: yaml.ValidatePartial 28 32 } 33 + -- out/yaml-v3 -- 34 + Errors: 35 + #test.t1.ok1: cannot call non-function fn (type _): 36 + ./in.cue:8:11 37 + validate.t1.ok1: invalid value "a: 2" (does not satisfy encoding/yaml.Validate({a!:int} | {b!:int})): error in call to encoding/yaml.Validate: incomplete value {a:2} | {a:2,b!:int}: 38 + ./in.cue:8:11 39 + ./in.cue:6:9 40 + #test.t1.ok2: cannot call non-function fn (type _): 41 + ./in.cue:9:11 42 + #test.t1.ok3: cannot call non-function fn (type _): 43 + ./in.cue:13:11 44 + #test.t2.ok1: cannot call non-function fn (type _): 45 + ./in.cue:17:11 46 + #test.t2.ok2: cannot call non-function fn (type _): 47 + ./in.cue:18:11 48 + 49 + Result: 50 + import "encoding/yaml" 51 + 52 + #test: { 53 + fn: _ 54 + data1: "a: 2" 55 + t1: { 56 + ok1: _|_ // #test.t1.ok1: cannot call non-function fn (type _) 57 + ok2: _|_ // #test.t1.ok2: cannot call non-function fn (type _) 58 + ok3: _|_ // #test.t1.ok3: cannot call non-function fn (type _) 59 + } 60 + #A: { 61 + a: int 62 + } 63 + #B: { 64 + b: int 65 + } 66 + data2: "'foo'" 67 + t2: { 68 + ok1: _|_ // #test.t2.ok1: cannot call non-function fn (type _) 69 + ok2: _|_ // #test.t2.ok2: cannot call non-function fn (type _) 70 + } 71 + } 72 + validate: { 73 + fn: yaml.Validate 74 + data1: "a: 2" 75 + 76 + // TODO: fix this test: the second disjunct should be eliminated, so there 77 + // should not be a concreteness error. 78 + t1: { 79 + ok1: _|_ // validate.t1.ok1: invalid value "a: 2" (does not satisfy encoding/yaml.Validate({a!:int} | {b!:int})): validate.t1.ok1: error in call to encoding/yaml.Validate: incomplete value {a:2} | {a:2,b!:int} 80 + ok2: "a: 2" 81 + ok3: "a: 2" 82 + } 83 + #A: { 84 + a: int 85 + } 86 + #B: { 87 + b: int 88 + } 89 + data2: "'foo'" 90 + t2: { 91 + ok1: "'foo'" 92 + ok2: "'foo'" 93 + } 94 + } 95 + validatePartial: { 96 + fn: yaml.ValidatePartial 97 + data1: "a: 2" 98 + t1: { 99 + ok1: "a: 2" 100 + ok2: "a: 2" 101 + ok3: "a: 2" 102 + } 103 + #A: { 104 + a: int 105 + } 106 + #B: { 107 + b: int 108 + } 109 + data2: "'foo'" 110 + t2: { 111 + ok1: "'foo'" 112 + ok2: "'foo'" 113 + } 114 + } 115 + -- diff/-out/yaml-v3<==>+out/yaml -- 116 + diff old new 117 + --- old 118 + +++ new 119 + @@ -4,7 +4,6 @@ 120 + validate.t1.ok1: invalid value "a: 2" (does not satisfy encoding/yaml.Validate({a!:int} | {b!:int})): error in call to encoding/yaml.Validate: incomplete value {a:2} | {a:2,b!:int}: 121 + ./in.cue:8:11 122 + ./in.cue:6:9 123 + - ./in.cue:7:16 124 + #test.t1.ok2: cannot call non-function fn (type _): 125 + ./in.cue:9:11 126 + #test.t1.ok3: cannot call non-function fn (type _): 127 + -- diff/todo -- 128 + missing position 29 129 -- out/yaml -- 30 130 Errors: 31 131 #test.t1.ok1: cannot call non-function fn (type _): 32 132 ./in.cue:8:11 133 + validate.t1.ok1: invalid value "a: 2" (does not satisfy encoding/yaml.Validate({a!:int} | {b!:int})): error in call to encoding/yaml.Validate: incomplete value {a:2} | {a:2,b!:int}: 134 + ./in.cue:8:11 135 + ./in.cue:6:9 136 + ./in.cue:7:16 33 137 #test.t1.ok2: cannot call non-function fn (type _): 34 138 ./in.cue:9:11 35 139 #test.t1.ok3: cannot call non-function fn (type _): ··· 65 169 validate: { 66 170 fn: yaml.Validate 67 171 data1: "a: 2" 172 + 173 + // TODO: fix this test: the second disjunct should be eliminated, so there 174 + // should not be a concreteness error. 68 175 t1: { 69 - ok1: fn({ 70 - a!: int 71 - } | { 72 - b!: int 73 - }) & data1 74 - ok2: fn(close({ 75 - a: int 76 - }) | close({ 77 - b: int 78 - })) & data1 79 - ok3: fn(#A | #B) & data1 176 + ok1: _|_ // validate.t1.ok1: invalid value "a: 2" (does not satisfy encoding/yaml.Validate({a!:int} | {b!:int})): validate.t1.ok1: error in call to encoding/yaml.Validate: incomplete value {a:2} | {a:2,b!:int} 177 + ok2: "a: 2" 178 + ok3: "a: 2" 80 179 } 81 180 #A: { 82 181 a: int ··· 86 185 } 87 186 data2: "'foo'" 88 187 t2: { 89 - ok1: fn(*int | string) & data2 90 - ok2: fn(string) & data2 188 + ok1: "'foo'" 189 + ok2: "'foo'" 91 190 } 92 191 } 93 192 validatePartial: { 94 193 fn: yaml.ValidatePartial 95 194 data1: "a: 2" 96 195 t1: { 97 - ok1: fn({ 98 - a!: int 99 - } | { 100 - b!: int 101 - }) & data1 102 - ok2: fn(close({ 103 - a: int 104 - }) | close({ 105 - b: int 106 - })) & data1 107 - ok3: fn(#A | #B) & data1 196 + ok1: "a: 2" 197 + ok2: "a: 2" 198 + ok3: "a: 2" 108 199 } 109 200 #A: { 110 201 a: int ··· 114 205 } 115 206 data2: "'foo'" 116 207 t2: { 117 - ok1: fn(*int | string) & data2 118 - ok2: fn(string) & data2 208 + ok1: "'foo'" 209 + ok2: "'foo'" 119 210 } 120 211 }