Add integration tests progress reporting #3471

rubiagatra · 2023-09-30T12:33:14Z

Description

Closes #3055 . Still WIP

Readiness checklist (Still in discussion)

I added/updated unit tests (and they pass).
I added/updated integration/compatibility tests (and they pass).
I added/updated comments and checked rendering.
I made spot refactorings.
I updated user documentation.
I ran task all, and it passed.
I ensured that PR title is good enough for the changelog.
(for maintainers only) I set Reviewers (@FerretDB/core), Milestone (Next), Labels, Project and project's Sprint fields.
I marked all done items in this checklist.

CLAassistant · 2023-09-30T12:33:19Z

All committers have signed the CLA.

AlekSi · 2023-09-30T12:35:04Z

@rubiagatra please leave a comment at #3055 so it can be assigned to you

rubiagatra · 2023-09-30T12:41:44Z

done @AlekSi! already commented

AlekSi

Please un-draft it and re-request review once it is ready

rubiagatra · 2023-10-07T09:32:17Z

Hey @AlekSi this is my first iteration sorry for still WIP. Could you check is this going in the right way?

The output of the integration should be like this

➜ task test-integration
task: [test-integration-pg] ../bin/envtool tests run --shard-index=1 --shard-total=1 --run='' -- -count=1 -timeout=35m -race=true -tags=ferretdb_debug,ferretdb_hana -shuffle=on -coverpkg=../... -coverprofile=integration-pg.txt . -target-backend=ferretdb-pg -target-tls -postgresql-url='postgres://username@127.0.0.1:5432/ferretdb?pool_max_conns=50&search_path=' -compat-url='mongodb://username:password@127.0.0.1:47018/?tls=true&tlsCertificateKeyFile=../build/certs/client.pem&tlsCaFile=../build/certs/rootCA-cert.pem' -disable-filter-pushdown=false -enable-sort-pushdown=false

Pass: github.com/FerretDB/FerretDB/integration/TestCommandsAdministrationServerStatusStress
Pass: github.com/FerretDB/FerretDB/integration/TestFindCommentMethod
Fail: github.com/FerretDB/FerretDB/integration/TestCreateOnInsertStressDiffCollection
Pass: github.com/FerretDB/FerretDB/integration/TestFindAndModifyCompatDotNotation/Conflict/TestFindAndModifyCompatDotNotation-Conflict_Scalars
Pass: github.com/FerretDB/FerretDB/integration/TestFindAndModifyCompatDotNotation/Conflict/TestFindAndModifyCompatDotNotation-Conflict_Int32s

I know it is not really ready to review but I need your guidance

Thank you!

AlekSi

I think the output should be something like that:

Pass: TestCommandsAdministrationServerStatusStress 1/1500
Pass: TestFindCommentMethod 2/1500
…

We could cut github.com/FerretDB/FerretDB/integration prefix and add current/total output.

cmd/envtool/envtool.go

rubiagatra · 2023-10-11T12:29:45Z

got it @AlekSi, let me update for it. I'll let you know if it ready

rubiagatra · 2023-10-11T13:34:05Z

Hi, @AlekSi I just updated the code It look like this

Pass: TestGetMoreCommandMaxTimeMSCursor/FindExpire 1/247
Pass: TestGetMoreCommandMaxTimeMSCursor/FindGetMoreMaxTimeMS 2/247
Pass: TestGetMoreCommandMaxTimeMSCursor/AggregateExpire 3/247

I know the total tests are not 247 because of nested tests. How we approach these problem? I only can get the root of the test

maybe want to sum all test inside integration test?

func countTestsInFile(filename string) int {
	content, err := ioutil.ReadFile(filename)
	if err != nil {
		return 0
	}

	re := regexp.MustCompile(`func Test\w+\(t \*testing\.T\)`)
	matches := re.FindAll(content, -1)
	return len(matches)
}

Taskfile.yml

cmd/envtool/testdata/error_test.go

cmd/envtool/tests.go

…eir-progress

rubiagatra · 2023-10-21T12:25:01Z

HI @AlekSi, Let me know if something is not right in my implementation at the error / skip test. I resolved all requested changes. Thank you!

…eir-progress

AlekSi · 2023-10-24T04:54:37Z

@rubiagatra I updated tests. I think we should include the output of the parent(s) test if subtest fail, so we could read the whole log in one place. On the other hand, if subtest fails, there is no need to show parent test output, I think.

And we should add a test where the parent test fails.

AlekSi

#3471 (comment)

rubiagatra · 2023-10-24T10:20:40Z

Got it @AlekSi will let you know if its done

…eir-progress

rubiagatra · 2023-10-30T03:14:29Z

HI @AlekSi Sorry for the late update. I a little bit struggle with this from:

                                FAIL TestError1 (1/2)
                                  === RUN   TestError1
                                  error_test.go:20: not hidden 1
                                  error_test.go:22: Error 1
                                  error_test.go:24: not hidden 2
                                  --- FAIL: TestError1 (0.00s)
                                PASS TestError2/NotParallel
                                FAIL TestError2/Parallel
                                  === RUN   TestError2/Parallel
                                  error_test.go:35: not hidden 5
                                  === PAUSE TestError2/Parallel
                                  === CONT  TestError2/Parallel
                                  error_test.go:39: not hidden 6
                                  error_test.go:41: Error 2
                                  error_test.go:43: not hidden 7
                                  --- FAIL: TestError2/Parallel (0.00s)
                                FAIL TestError2 (2/2)
                                  === RUN   TestError2
                                  error_test.go:28: not hidden 3
                                  === PAUSE TestError2
                                  === CONT  TestError2
                                  error_test.go:32: not hidden 4
                                  --- FAIL: TestError2 (0.00s)

                      "FAIL TestError1 (1/2)",
                      "  === RUN   TestError1",
                      "  error_test.go:20: not hidden 1",
                      "  error_test.go:22: Error 1",
                      "  error_test.go:24: not hidden 2",
                      "  --- FAIL: TestError1 (0.00s)",
                      "PASS TestError2/NotParallel",
                      "FAIL TestError2/Parallel",
                      "  === RUN   TestError2/Parallel",
                      "  error_test.go:28: not hidden 3",
                      "  === PAUSE TestError2",
                      "  === CONT  TestError2",
                      "  error_test.go:32: not hidden 4",
                      "  error_test.go:35: not hidden 5",
                      "  === PAUSE TestError2/Parallel",
                      "  === CONT  TestError2/Parallel",
                      "  error_test.go:39: not hidden 6",
                      "  error_test.go:41: Error 2",
                      "  error_test.go:43: not hidden 7",
                      "  --- FAIL: TestError2/Parallel (0.00s)",
                      "FAIL TestError2 (2/2)",

Can you give me an idea of how we do this?

Should we collect all the logs first, check which child failed, and log them all in the correct order based on your suggestion? We cannot print directly, as we do now.

Thank you!

AlekSi · 2023-10-30T15:32:16Z

@rubiagatra I played a bit with the expected output to make it good enough with a simple implementation – and I think it is good enough now :) Thank you!

rubiagatra · 2023-10-31T00:37:43Z

I want to say thank you for your support @AlekSi, I hope I can contribute more in the future!

rubiagatra requested a review from a team as a code owner September 30, 2023 12:33

rubiagatra requested review from AlekSi and chilagrow September 30, 2023 12:33

mergify bot assigned rubiagatra Sep 30, 2023

rubiagatra marked this pull request as draft September 30, 2023 12:33

rubiagatra force-pushed the test/3055-integration-tests-should-report-their-progress branch from fae6ad6 to 5d66c52 Compare September 30, 2023 12:41

AlekSi removed request for AlekSi and chilagrow October 4, 2023 05:06

AlekSi added the code/chore Code maintenance improvements label Oct 4, 2023

AlekSi added this to the Next milestone Oct 4, 2023

AlekSi reviewed Oct 4, 2023

View reviewed changes

rubiagatra force-pushed the test/3055-integration-tests-should-report-their-progress branch from 5d66c52 to 241cafd Compare October 7, 2023 09:28

rubiagatra changed the title ~~draft: start to investigate report of integration test~~ test: start to investigate report of integration test Oct 7, 2023

rubiagatra requested a review from AlekSi October 7, 2023 09:29

rubiagatra marked this pull request as ready for review October 7, 2023 09:30

rubiagatra force-pushed the test/3055-integration-tests-should-report-their-progress branch from 241cafd to 668e1bc Compare October 7, 2023 09:34

AlekSi modified the milestones: v1.12.0, Next Oct 9, 2023

AlekSi self-assigned this Oct 10, 2023

AlekSi reviewed Oct 10, 2023

View reviewed changes

cmd/envtool/envtool.go Outdated Show resolved Hide resolved

cmd/envtool/envtool.go Outdated Show resolved Hide resolved

rubiagatra force-pushed the test/3055-integration-tests-should-report-their-progress branch from 668e1bc to ff897de Compare October 11, 2023 12:28

rubiagatra force-pushed the test/3055-integration-tests-should-report-their-progress branch 3 times, most recently from 458dbb0 to 22d28f7 Compare October 11, 2023 13:33

AlekSi reviewed Oct 19, 2023

View reviewed changes

Taskfile.yml Outdated Show resolved Hide resolved

AlekSi reviewed Oct 19, 2023

View reviewed changes

cmd/envtool/testdata/error_test.go Show resolved Hide resolved

AlekSi reviewed Oct 19, 2023

View reviewed changes

cmd/envtool/tests.go Outdated Show resolved Hide resolved

AlekSi reviewed Oct 19, 2023

View reviewed changes

cmd/envtool/tests.go Outdated Show resolved Hide resolved

rubiagatra added 2 commits October 19, 2023 20:14

Merge branch 'main' into test/3055-integration-tests-should-report-th…

6253420

…eir-progress

fix requested changes, add skip, and log

d69129e

mergify bot removed the conflict PRs that have merge conflicts label Oct 20, 2023

rubiagatra requested a review from AlekSi October 20, 2023 12:50

AlekSi added 2 commits October 21, 2023 22:25

Merge branch 'main' into test/3055-integration-tests-should-report-th…

6a92412

…eir-progress

Merge branch 'main' into test/3055-integration-tests-should-report-th…

de1bb12

…eir-progress

AlekSi modified the milestones: v1.13.0, Next Oct 23, 2023

AlekSi added 2 commits October 24, 2023 08:38

Merge branch 'main' into test/3055-integration-tests-should-report-th…

019400f

…eir-progress

Update tests

2036ff7

AlekSi reviewed Oct 24, 2023

View reviewed changes

rubiagatra added 2 commits October 25, 2023 08:24

Merge branch 'main' into test/3055-integration-tests-should-report-th…

6535bc8

…eir-progress

Merge branch 'main' into test/3055-integration-tests-should-report-th…

b7cd211

…eir-progress

AlekSi added 6 commits October 30, 2023 15:03

Change the way the output is grouped

8fcc5d8

Do not log passing subtests

23f3d93

Tweak logging

f6a1c03

Merge branch 'main' into pr/rubiagatra/3471

e5396e1

Lint

245e264

Tweak logging

d1d8711

AlekSi merged commit 09ec58c into FerretDB:main Oct 30, 2023
25 of 26 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add integration tests progress reporting #3471

Add integration tests progress reporting #3471

rubiagatra commented Sep 30, 2023 •

edited

Loading

CLAassistant commented Sep 30, 2023 •

edited

Loading

AlekSi commented Sep 30, 2023

rubiagatra commented Sep 30, 2023

AlekSi left a comment

rubiagatra commented Oct 7, 2023 •

edited

Loading

AlekSi left a comment

rubiagatra commented Oct 11, 2023

rubiagatra commented Oct 11, 2023 •

edited

Loading

rubiagatra commented Oct 21, 2023

AlekSi commented Oct 24, 2023

AlekSi left a comment

rubiagatra commented Oct 24, 2023

rubiagatra commented Oct 30, 2023

AlekSi commented Oct 30, 2023

rubiagatra commented Oct 31, 2023

Add integration tests progress reporting #3471

Add integration tests progress reporting #3471

Conversation

rubiagatra commented Sep 30, 2023 • edited Loading

Description

Readiness checklist (Still in discussion)

CLAassistant commented Sep 30, 2023 • edited Loading

AlekSi commented Sep 30, 2023

rubiagatra commented Sep 30, 2023

AlekSi left a comment

Choose a reason for hiding this comment

rubiagatra commented Oct 7, 2023 • edited Loading

AlekSi left a comment

Choose a reason for hiding this comment

rubiagatra commented Oct 11, 2023

rubiagatra commented Oct 11, 2023 • edited Loading

rubiagatra commented Oct 21, 2023

AlekSi commented Oct 24, 2023

AlekSi left a comment

Choose a reason for hiding this comment

rubiagatra commented Oct 24, 2023

rubiagatra commented Oct 30, 2023

AlekSi commented Oct 30, 2023

rubiagatra commented Oct 31, 2023

rubiagatra commented Sep 30, 2023 •

edited

Loading

CLAassistant commented Sep 30, 2023 •

edited

Loading

rubiagatra commented Oct 7, 2023 •

edited

Loading

rubiagatra commented Oct 11, 2023 •

edited

Loading