Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

renaming part of extension build refactor PR #7926

Merged

Conversation

samansmink
Copy link
Contributor

This PR

this pr contains some makefile renames and file renames that are required for #7735
this is pr-ed separately into master to prevent causing lots of merge conflicts while having master and feature separate.

and a bit of ✂️🦬

This PR also contains some yak shaving in the form of introducing a "poor mans mono-repo" feature. It is achieved through the .github/patches mechanism that @carlopi and me thought of yesterday. The idea is that when a change is made to duckdb that breaks compatibility with duckdb-wasm, we can still have duckdb CI pass by adding the required changes to duckdb-wasm as a patch file.

This creates a natural flow for making changes to duckdb that looks like this:

  1. Create breaking change in duckdb
  2. Fix resulting issues in duckdb-wasm, store patch in .github/patches/duckdb-wasm with descriptive name.
  3. PR to duckdb the breaking change + patch
  4. PR to duckdb-wasm the changes from the (potentially multiple) patch(es)
  5. PR to duckdb a bump of the duckdb-wasm version pinned in .github/workflows/WebAssembly.yml along with the removal of the applied patches in .github/patches/duckdb-wasm

The nice thing is that CI will be able to pass during every step of this flow. Also, since wasm has its own versions and duckdb's version pinned applying the patch(es) to wasm does not need to happen for every breaking change, only when bumping the duckdb version in duckdb-wasm (though these patches should probably be rare and not be allowed to accumulate too much anyway).

@Mytherin interested to hear your thoughts! we could also apply a similar approach to out of tree extensions that are built using DuckDB's main CI, thought it is slightly different as deployment is then done through duckdb ci creating a bit more potential to "forget" to apply patches.

@samansmink samansmink requested a review from Mytherin June 13, 2023 07:54
@carlopi
Copy link
Contributor

carlopi commented Jun 13, 2023

I think this is a nice improvement over the current break-and-fix approach in synchronising repositories, since it reduces the amount of coordination required, and it's atomic (as explained by Sam, steps 3, 4, and 5 have to be completed in order but states will all be valid on both repos).

Note also that multiple parallel API-breaking PRs are also supported, as long as patches can be independently applied.

Proposal would be giving a test-run with DuckDB-Wasm, where this tries to address a present pain-point, can then be expanded to out-of-tree extensions in a separate step. This could also allow to eventually migrate (promote?) more extension to out-of-tree, given that this provides an arguably simpler workflow in adapting to eventual changes.

Makefile Outdated Show resolved Hide resolved
Makefile Outdated Show resolved Hide resolved
@Mytherin Mytherin merged commit f93e1c4 into duckdb:master Jun 15, 2023
@Mytherin
Copy link
Collaborator

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants