Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Out of bounds panic on YAML parsing #97651

Closed
howardjohn opened this issue Jan 2, 2021 · 9 comments · Fixed by #97990
Closed

Out of bounds panic on YAML parsing #97651

howardjohn opened this issue Jan 2, 2021 · 9 comments · Fixed by #97990
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. triage/accepted Indicates an issue or PR is ready to be actively worked on.

Comments

@howardjohn
Copy link
Contributor

This bug was privately disclosed and told to open here.

What happened:

NewYAMLOrJSONDecoder from k8s.io/apimachinery/pkg/util/yaml has an index out of bounds issue which can cause callers to panic.

Download attached reproducer file
Run golang test:

[repro.txt](https://github.com/kubernetes/kubernetes/files/5759584/repro.txt)

func TestCrash(t *testing.T) {
    f, _ := ioutil.ReadFile("repro")
    decoder := kubeyaml.NewYAMLOrJSONDecoder(bytes.NewBuffer(f), 1024)
    for {
        into := &struct{}{}
        err := decoder.Decode(&into)
        if err == io.EOF {
            break
        }
    }
}

When run with k8s.io/apimachinery v0.19.3:

=== RUN   TestCrash
--- FAIL: TestCrash (0.00s)
panic: runtime error: slice bounds out of range [:524297] with length 1024 [recovered]
    panic: runtime error: slice bounds out of range [:524297] with length 1024

goroutine 26 [running]:
testing.tRunner.func1.1(0x1ac7380, 0xc00090dc00)
    /usr/local/go/src/testing/testing.go:940 +0x2f5
testing.tRunner.func1(0xc000b4a120)
    /usr/local/go/src/testing/testing.go:943 +0x3f9
panic(0x1ac7380, 0xc00090dc00)
    /usr/local/go/src/runtime/panic.go:969 +0x166
k8s.io/apimachinery/pkg/util/yaml.(*YAMLOrJSONDecoder).Decode(0xc000529f30, 0x177b980, 0xc0000106c8, 0x0, 0x0)
    /home/howardjohn/go/pkg/mod/k8s.io/apimachinery@v0.19.3/pkg/util/yaml/decoder.go:245 +0x620
istio.io/istio/tests/fuzz.TestCrash(0xc000b4a120)
    /home/howardjohn/go/src/istio.io/istio/tests/fuzz/regression_test.go:22 +0x12d
testing.tRunner(0xc000b4a120, 0x1cebf78)
    /usr/local/go/src/testing/testing.go:991 +0xdc
created by testing.(*T).Run
    /usr/local/go/src/testing/testing.go:1042 +0x357

Additionally, this can be seen in kubectl directly (below is kubectl 1.20.0):

$ kubectl apply -f  repro --dry-run=client --validate=false
panic: runtime error: slice bounds out of range [:526607] with length 4096

goroutine 1 [running]:
k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/yaml.(*YAMLOrJSONDecoder).Decode(0xc0005af6d0, 0x1a3d3e0, 0xc00094d7a0, 0x0, 0x0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/yaml/decoder.go:276 +0x67d
k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource.(*StreamVisitor).Visit(0xc000428900, 0xc000997ea0, 0x1000, 0xc0009a5000)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:576 +0xb9
k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource.(*FileVisitor).Visit(0xc000997da0, 0xc000997ea0, 0x0, 0x0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:528 +0x25d
k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource.EagerVisitorList.Visit(0xc0030f8740, 0x1, 0x1, 0xc000428a80, 0xc000428a01, 0xc000428a80)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:213 +0xc5
k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource.FlattenListVisitor.Visit(0x1cee740, 0xc000997dc0, 0x1cf9780, 0x2a9d9d0, 0xc0009a2180, 0xc000428a00, 0x10000c0030f8730, 0xc0005af8f8)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:391 +0x86
k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource.FlattenListVisitor.Visit(0x1cee780, 0xc0009a21b0, 0x1cf9780, 0x2a9d9d0, 0xc0009a2180, 0xc000997e60, 0x40d930, 0xc0005af9a8)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:391 +0x86
k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource.ContinueOnErrorVisitor.Visit(0x1cee780, 0xc0009a2210, 0xc0004289c0, 0xc0030f8701, 0xc0004289c0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:354 +0x9c
k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource.DecoratedVisitor.Visit(0x1cee700, 0xc0030f8770, 0xc000997e00, 0x3, 0x4, 0xc0030f8780, 0xc00095d680, 0xc0005afaf8)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource/visitor.go:326 +0x86
k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource.(*Result).Infos(0xc00095d680, 0xc00095d680, 0x0, 0xc0007e0000, 0x2a9d9d0, 0xc0005afbb0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/cli-runtime/pkg/resource/result.go:122 +0xc2
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/apply.(*ApplyOptions).GetObjects(0xc000696820, 0xc0030f86c0, 0xc0005afbd0, 0x160f02d, 0x1922840, 0xc0030f86c0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/apply/apply.go:342 +0x185
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/apply.(*ApplyOptions).Run(0xc000696820, 0x0, 0x1bb70f8)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/apply/apply.go:373 +0x5e
k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/apply.NewCmdApply.func1(0xc000960000, 0xc00041ac00, 0x0, 0x4)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/k8s.io/kubectl/pkg/cmd/apply/apply.go:180 +0x107
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).execute(0xc000960000, 0xc00041abc0, 0x4, 0x4, 0xc000960000, 0xc00041abc0)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:854 +0x2c2
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc0001c8b00, 0xc000184180, 0xc000184120, 0x6)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:958 +0x375
k8s.io/kubernetes/vendor/github.com/spf13/cobra.(*Command).Execute(...)
        /workspace/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/vendor/github.com/spf13/cobra/command.go:895
main.main()
        _output/dockerized/go/src/k8s.io/kubernetes/cmd/kubectl/kubectl.go:49 +0x21d

Environment:

  • Kubernetes version (use kubectl version): Verified with 1.19, 1.20

repro.txt

@howardjohn howardjohn added the kind/bug Categorizes issue or PR as related to a bug. label Jan 2, 2021
@k8s-ci-robot k8s-ci-robot added needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 2, 2021
@howardjohn
Copy link
Contributor Author

/sig api-machinery

@k8s-ci-robot k8s-ci-robot added sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Jan 2, 2021
@AlexanderYastrebov
Copy link
Contributor

AlexanderYastrebov commented Jan 2, 2021

This piece

if syntax, ok := err.(*json.SyntaxError); ok {
data, readErr := ioutil.ReadAll(jsonDecoder.Buffered())
if readErr != nil {
klog.V(4).Infof("reading stream failed: %v", readErr)
}
js := string(data)
// if contents from io.Reader are not complete,
// use the original raw data to prevent panic
if int64(len(js)) <= syntax.Offset {
js = string(d.rawData)
}
start := strings.LastIndex(js[:syntax.Offset], "\n") + 1
line := strings.Count(js[:start], "\n")
return JSONSyntaxError{
Line: line,
Err: fmt.Errorf(syntax.Error()),
}
}

attempts to find line+column position of a json syntax error.

To do that it assumes that full input is available which is not true:

  1. jsonDecoder.Buffered() is not a full input:

Buffered returns a reader of the data remaining in the Decoder's buffer. The reader is valid until the next call to Decode.

  1. d.rawData is only the first data chunk:
    buffer, origData, isJSON := GuessJSONStream(d.r, d.bufferSize)
    if isJSON {
    d.decoder = json.NewDecoder(buffer)
    d.rawData = origData
    } else {

    func GuessJSONStream(r io.Reader, size int) (io.Reader, []byte, bool) {
    buffer := bufio.NewReaderSize(r, size)
    b, _ := buffer.Peek(size)
    return buffer, b, hasJSONPrefix(b)
    }

The subject happens e.g. when syntax error offset is past buffer size and int64(len(js)) <= syntax.Offset
Test to reproduce the panic:

diff --git a/staging/src/k8s.io/apimachinery/pkg/util/yaml/decoder_test.go b/staging/src/k8s.io/apimachinery/pkg/util/yaml/decoder_test.go
index 1f4b158f968..a60027a05ce 100644
--- a/staging/src/k8s.io/apimachinery/pkg/util/yaml/decoder_test.go
+++ b/staging/src/k8s.io/apimachinery/pkg/util/yaml/decoder_test.go
@@ -247,6 +247,15 @@ func TestDecodeBrokenJSON(t *testing.T) {
        }
 }
 
+func Test97651(t *testing.T) {
+    s := NewYAMLOrJSONDecoder(bytes.NewReader([]byte(`{
+       "foo": {
+               "stuff": 1,
+}`)), 1)
+       obj := generic{}
+       s.Decode(&obj)
+}
+
 type generic map[string]interface{}
 
 func TestYAMLOrJSONDecoder(t *testing.T) {

I am not sure it is easily possible to find line+column of json syntax error as full input may not be available.
Maybe it should only attempt to find line+column if offset is within the first chunk (i.e. below buffer size).

@AlexanderYastrebov
Copy link
Contributor

Line number calculation added in #25038 and updated in #38525

@dims
Copy link
Member

dims commented Jan 8, 2021

@AlexanderYastrebov do you want to submit a patch to decoder.do with a defensive check to avoid the panic ( compare len(js) and syntax.Offset)

@AlexanderYastrebov
Copy link
Contributor

@dims
I have raised a discussion golang/go#43513 on having line number in the json.SyntaxError. The consensus there is that it should be possible to count lines with a full input at hand and it does not worth to have it in the core library due to performance impact and maintenance costs. The problem of YAMLOrJSONDecoder here is that it attempts to find out line number without having full input at hand.

I think the defensive patch is not the right approach because line calculation is incorrect anyway even when there is no panic: jsonDecoder.Buffered() begins at a value with a syntax error, i.e. in a stream of two values and syntax error in the second value it would contain only the second value and thus new lines in the first value in the stream would not be counted.

I think the following options are there:

  1. Change JSONSyntaxError Line to stream Offset taken from json.SyntaxError. This is straightforward but changes JSONSyntaxError interface. I can not find any code that uses JSONSyntaxError.Line field, it is only used in the error message that would change:
    func (e JSONSyntaxError) Error() string {
    return fmt.Sprintf("json: line %d: %s", e.Line, e.Err.Error())
    }
  2. Calculate line number only if offset is within the length of the first data chunk (d.rawData). I am not sure what would be the unknown line number then, -1 perhaps.
  3. Calculate line number properly during stream decoding, see encoding/json: add line number to SyntaxError golang/go#43513 (comment) This has some performance penalty as line number is calculated in any case even if there is no syntax error, i.e. in the majority of the cases.

@caesarxuchao
Copy link
Member

cc @liggitt @lavalamp
/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jan 12, 2021
@lavalamp
Copy link
Member

Change JSONSyntaxError Line to stream Offset taken from json.SyntaxError. This is straightforward but changes JSONSyntaxError interface. I can not find any code that uses JSONSyntaxError.Line field, it is only used in the error message that would change:

This is probably acceptable.

@AlexanderYastrebov
Copy link
Contributor

@lavalamp

This is probably acceptable.

I have opened a PR

@AlexanderYastrebov
Copy link
Contributor

@lavalamp Thank you for support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/bug Categorizes issue or PR as related to a bug. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. triage/accepted Indicates an issue or PR is ready to be actively worked on.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants