[CI] More CI reduction and clean-up #8052

Mytherin · 2023-06-22T20:38:38Z

Follow-up from #8046 (also includes that PR)

Nightly Test

Add a NightlyTest.yml that contains several tests that will not run on every PR, but only during the nightly build. This includes several tests that rarely fail, but where it is still a good idea that we run them semi-frequently.

The nightly tests contain the following tests:

Linux Memory Leaks
Release Assertions
Thread Sanitizer
Vector Size = 2
Clang 14
GCC 4.8
Python 3 Tarball
Python Address Sanitizer (Linux)
Sqllogic tests
Storage Initialization Verification
No String Inline / Destroy Unpinned Blocks
Forcing async Sinks/Sources
Regression Tests between safe and unsafe builds
R Package Windows (Warnings)
R UBSAN

The nightly tests are still run in the CI of the forks, so for bigger PRs/refactors it is still possible to verify that they pass by checking there.

Test All Types

By default test_all_types no longer emits a giant enum with 70K+ values for large_enum - as this ends up taking a long time in debug mode making tests that use the test_all_types function take a good amount of time (even if the enum is never used). Instead there is now a use_large_enum option that can be set to actually generate the large enum.

Note that the data that is generated is the same, only the underlying enum type changes based on this option (containing either enum_0, enum_69999 or enum_0, enum_1, ..., enum_69998, enum_69999).

Slow Tests

This PR also moves several tests to slow tests and removes some unnecessary/excessive loops in tests. We also add a profiling mode to run_tests_one_by_one, allowing the tests to be profiled so we can easily find slow tests in the future.

python3 scripts/run_tests_one_by_one.py build/debug/test/unittest --profile

R Extensions

Previously we had a special test for building and loading extensions in R. This was required/useful since we had special handling for dynamic loading of extensions in R. Ever since the static linking this is rather moot since the extension loading for R is not special anymore - yet the CI run still added a significant amount of time. This PR removes the CI run and the R extension loading tests.

Other Changes

Remove the Extended test. This test is not necessary since other tests (tidy check/windows) already build with DISABLE_UNITY.
Merge the Windows ODBC and Windows 64-bit tests. The Windows ODBC test took 50 minutes, of which 48 minutes are building. Moving the ODBC tests to Windows 64-bit speeds up the CI significantly.

This reverts commit 70bf771.

…dows

This reverts commit ce459c2.

…ons and old GCC version there

carlopi

I have a comment I would like to see improved (potentially on a separate iteration), but apart from that LGTM!

I haven't had a real try at how the workflow feels on a fork, but any problem could be fixed up later in a second part together with Swift workflow (it's the only failure, I would be ok with merging AND temporarily disabling Swift.yml).

Of the in-flight PRs I believe #7735, that has all test passing, would be most likely be incompatible, so I would merge first either this or #7735, and then fixing up whoever goes in as second.

carlopi · 2023-06-23T06:53:16Z

.github/workflows/LinuxRelease.yml

@@ -211,6 +212,7 @@ jobs:

 linux-extensions-64-aarch64:
    name: Linux Extensions (AARCH64)
+    if: startsWith(github.ref, 'refs/tags/v') || github.ref == 'refs/heads/master'


Suggested change

if: startsWith(github.ref, 'refs/tags/v') || github.ref == 'refs/heads/master'

if: startsWith(github.ref, 'refs/tags/v') || github.ref == 'refs/heads/master' || github.repo != 'duckdb/duckdb'

(syntax is very much pseudo-code)
I think we basically want to restrict this (and a few other similar places) only for duckdb/duckdb, since it would be useful to have this run also from any branch on forks.

Merge pull request duckdb/duckdb#8307 from Tishj/chunk_scan_state [Arrow] Add ChunkScanState interface to preserve chunk-offset when scanning Merge pull request duckdb/duckdb#8089 from pdet/basepython_tz Python TIMESTAMPTZ support Merge pull request duckdb/duckdb#8052 from Mytherin/evenlessci [CI] More CI reduction and clean-up Merge pull request duckdb/duckdb#7701 from Mause/bugfix/python-union-description Add union to test_all_types, and arrow and json R/W

- Merge pull request duckdb/duckdb#8307 from Tishj/chunk_scan_state: [Arrow] Add ChunkScanState interface to preserve chunk-offset when scanning - Merge pull request duckdb/duckdb#8089 from pdet/basepython_tz: Python TIMESTAMPTZ support - Merge pull request duckdb/duckdb#8052 from Mytherin/evenlessci: [CI] More CI reduction and clean-up - Merge pull request duckdb/duckdb#7701 from Mause/bugfix/python-union-description: Add union to test_all_types, and arrow and json R/W - Merge pull request duckdb/duckdb#8497 from samansmink/pending-execute-result-api-change: Add PendingExecutionResult::ALL_TASKS_BLOCKED

Mytherin added 30 commits June 22, 2023 10:58

make coverage-check

bb966b8

Skip more CI on non-releases

0cb9d31

Skip more jobs

93927a0

Switch to ubuntu-latest

70bf771

Add the other tests back in

2884985

Revert "Switch to ubuntu-latest"

d04892e

This reverts commit 70bf771.

More ifs

5a61e6c

Run this in bash

366056b

Comments

a9997f8

Windows - unify ODBC test with main Windows test, and skip 32-bit win…

f665d46

…dows

Move release assertions to extra tests

ce459c2

Revert "Move release assertions to extra tests"

ed7d423

This reverts commit ce459c2.

Add extra NightlyTests that are run nightly, and move release asserti…

84580d9

…ons and old GCC version there

Move more tests to Nightly

cbf2787

Remove Expanded test - it is superseded by tidy-check

6adaef9

Move sqllogic test to nightly test

f3a9959

Move more workflows to nightly tests

8df9f84

Remove R extension loading test (no longer required)

67d1269

YAML yay

b575988

No node ARM on PRs

e969c19

Remove needs:

72ec504

needs: to avoid running tons of jobs

2c1c7a9

Move R tests to nightly tests

8a16d76

test_all_types - strip large_enum

861cbf9

Test clean-up - move several tests to slow tests

0add37a

Add use_large_enum option to test_all_types

1ce0f2a

Format

8ca3b7c

use_large_enum for R

0b9034f

Add --profile option to run_tests_one_by_one

e996f45

Move vector sizes and thread sanitizer to nightly tests

d7c2923

Mytherin added 4 commits June 22, 2023 19:39

More needs:

5b53641

But why

e880c79

Increment uncovered_files after moving tests to slow

17b1906

large_enum=true

ec21512

Mytherin closed this Jun 23, 2023

Mytherin reopened this Jun 23, 2023

carlopi approved these changes Jun 23, 2023

View reviewed changes

Swift fixes

ab7e651

Mytherin marked this pull request as draft June 23, 2023 07:08

Mytherin marked this pull request as ready for review June 23, 2023 07:08

use_large_enum

9513536

Mytherin marked this pull request as draft June 23, 2023 08:17

Mytherin marked this pull request as ready for review June 23, 2023 08:18

Mytherin merged commit 155b1b4 into duckdb:master Jun 23, 2023

Mytherin deleted the evenlessci branch July 5, 2023 03:39

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] More CI reduction and clean-up #8052

[CI] More CI reduction and clean-up #8052

Mytherin commented Jun 22, 2023

carlopi left a comment

carlopi Jun 23, 2023

	if: startsWith(github.ref, 'refs/tags/v') \|\| github.ref == 'refs/heads/master'
	if: startsWith(github.ref, 'refs/tags/v') \|\| github.ref == 'refs/heads/master' \|\| github.repo != 'duckdb/duckdb'

[CI] More CI reduction and clean-up #8052

[CI] More CI reduction and clean-up #8052

Conversation

Mytherin commented Jun 22, 2023

Nightly Test

Test All Types

Slow Tests

R Extensions

Other Changes

carlopi left a comment

Choose a reason for hiding this comment

carlopi Jun 23, 2023

Choose a reason for hiding this comment