text/scanner: SkipComments is ignored without ScanComments #71133
Open
Description
Go version
go1.23.4
Output of go env in your module/workspace:
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/carlo/.cache/go-build'
GOENV='/home/carlo/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/carlo/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/carlo/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/lib/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/lib/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.23.4'
GODEBUG=''
GOTELEMETRY='off'
GOTELEMETRYDIR='/home/carlo/.config/go/telemetry'
GCCGO='gccgo'
GOAMD64='v3'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/carlo/src/go-scanner-bug/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build186288891=/tmp/go-build -gno-record-gcc-switches'What did you do?
The text/scanner docs say:
Predefined mode bits to control recognition of tokens. For instance, to configure a Scanner such that it only recognizes (Go) identifiers, integers, and skips comments, set the Scanner's Mode field to:
ScanIdents | ScanInts | SkipCommentsWith the exceptions of comments, which are skipped if SkipComments is set, unrecognized tokens are not ignored.
However, it seems that SkipComments only works as advertised when combined with ScanComments, despite the example clearly implying otherwise.
The following code demonstrates (see go.dev.play link):
package main
import (
"fmt"
"strings"
"text/scanner"
)
func main() {
testScanner(scanner.ScanIdents|scanner.ScanInts|scanner.SkipComments, "// comment")
testScanner(scanner.ScanIdents|scanner.ScanInts|scanner.ScanComments|scanner.SkipComments, "// comment")
}
func testScanner(mode uint, input string) {
var sc scanner.Scanner
sc.Init(strings.NewReader(input))
sc.Mode = mode
for sc.Peek() != scanner.EOF {
tok := sc.Scan()
fmt.Printf("[%s:'%s'] ", scanner.TokenString(tok), sc.TokenText())
}
fmt.Println()
}What did you see happen?
The output is:
["/":'/'] ["/":'/'] [Ident:'comment']
[EOF:'']
What did you expect to see?
I would expect the output to be:
[EOF:'']
[EOF:'']