Revert "Revert "Support macros when cross-compiling (#7118)" (#7352)" #7353

MaxDesiatov · 2024-02-20T20:15:00Z

Reverts #7352.

Modified to build tests for the host triple when any macros are added to test modules directly as dependencies.

This reverts commit adfbd9a.

MaxDesiatov · 2024-02-20T20:15:07Z

@swift-ci test

MaxDesiatov · 2024-02-21T11:03:55Z

Currently blocked by CI failures in swiftlang/swift-syntax#2504

MaxDesiatov · 2024-02-26T13:37:22Z

@swift-ci test

MaxDesiatov · 2024-02-26T15:07:39Z

@swift-ci test windows

…vert-7352-maxd/revert-cross-compilation # Conflicts: # Sources/Build/BuildDescription/SwiftTargetBuildDescription.swift # Sources/Build/BuildManifest/LLBuildManifestBuilder.swift # Sources/Build/BuildPlan/BuildPlan+Test.swift # Sources/Build/BuildPlan/BuildPlan.swift # Sources/Commands/PackageCommands/PluginCommand.swift # Sources/Commands/SwiftTestCommand.swift # Sources/Commands/Utilities/PluginDelegate.swift # Sources/Commands/Utilities/TestingSupport.swift # Sources/SPMTestSupport/MockBuildTestHelper.swift # Sources/SPMTestSupport/MockPackageGraphs.swift # Sources/SPMTestSupport/PackageGraphTester.swift # Tests/BuildTests/BuildPlanTests.swift # Tests/PackageGraphTests/ModulesGraphTests.swift # Tests/SourceKitLSPAPITests/SourceKitLSPAPITests.swift

MaxDesiatov · 2024-03-08T12:19:21Z

@swift-ci test

…vert-7352-maxd/revert-cross-compilation

…lation`" This reverts commit 10c7cd1.

MaxDesiatov · 2024-03-20T19:17:57Z

@swift-ci test

MaxDesiatov · 2024-03-20T23:14:52Z

@swift-ci test

MaxDesiatov · 2024-03-21T21:43:42Z

@swift-ci test

MaxDesiatov · 2024-03-25T18:22:34Z

@swift-ci test

Sources/Build/BuildDescription/ResolvedTarget+BuildDescription.swift

grynspan · 2024-03-31T14:59:11Z

@swift-ci please test Windows

MaxDesiatov · 2024-04-16T10:14:01Z

@swift-ci test

Benchmarking modules graph is now generalized with `syntheticModulesGraph` function. It also uncovered a bug in the previous `SyntheticModulesGraph`, which didn't pass generated `TargetDescription`s array to `loadModulesGraph`, which is fixed now. New `SyntheticModulesGraphWithMacros` calls `syntheticModulesGraph` with `includeMacros: true` argument, which splits all modules in three parts: library modules, macros modules that library modules depend on, and macro dependencies that macros depend on. This allows us to track potential performance regressions in #7353.

MaxDesiatov · 2024-04-17T17:47:42Z

@swift-ci test windows

rauhul · 2024-04-17T21:08:36Z

EXCITING :D

Benchmarking modules graph is now generalized with `syntheticModulesGraph` function. It also uncovered a bug in the previous `SyntheticModulesGraph`, which didn't pass generated `TargetDescription`s array to `loadModulesGraph`, which is fixed now. New `SyntheticModulesGraphWithMacros` calls `syntheticModulesGraph` with `includeMacros: true` argument, which splits all modules in three parts: library modules, macros modules that library modules depend on, and macro dependencies that macros depend on. This allows us to track potential performance regressions in #7353.

finagolfin · 2024-04-25T14:41:43Z

@MaxDesiatov, thanks for working on this, looking forward to using it.

I tried using the official April 22 snapshot for ubi9 x86_64 to cross-compile the swift-syntax macro examples, after patching swift-build to apply your fix from #7493, but I'm seeing errors like this:

> cd swift-syntax/Examples/
> ~/swift-DEVELOPMENT-SNAPSHOT-2024-04-22-a-ubi9/usr/bin/swift build --build-tests --destination ~/swift-android-sdk/android-aarch64.json -Xlinker -rpath -Xlinker \$ORIGIN/swift-trunk-android-aarch64-2024-04-22-24-sdk/usr/lib/swift/android
Building for debugging...
error: Failed opening '/home/fina/swift-syntax/Examples/.build/aarch64-unknown-linux-android28/debug/index/store/v5/units/AddAsyncMacroTests.swift.o-1VZ0WNZLXJYVM': No such file or directory

Swift-foundation also doesn't build with similar errors, but other macro-free packages cross-compile fine. Are you seeing build errors like this when cross-compiling packages that have macros?

After #7353 landed, I noticed that the build products for test targets were not being emitted correctly. swift-testing and XCTest produce separate build products (with distinct names) but this wasn't happening as intended. It turns out that the changes to split `buildParameters` into `productsBuildParameters` and `toolsBuildParameters` weren't fully propagated to our testing infrastructure. I also noticed `SWIFT_PM_SUPPORTS_SWIFT_TESTING` wasn't being set correctly anymore (same root cause) although we've decided to ignore that flag over in swift-testing anyway (see swiftlang/swift-testing#376.) This regression caused build failures in swift-testing (e.g. [here](https://ci.swift.org/job/pr-swift-testing-macos/663/console)) with the telltale failure signature: > /Users/ec2-user/jenkins/workspace/pr-swift-testing-macos/branch-main/swift-testing/.build/x86_64-apple-macosx/debug/swift-testingPackageTests.xctest/Contents/MacOS/swift-testingPackageTests: /Users/ec2-user/jenkins/workspace/pr-swift-testing-macos/branch-main/swift-testing/.build/x86_64-apple-macosx/debug/swift-testingPackageTests.xctest/Contents/MacOS/swift-testingPackageTests: cannot execute binary file Which indicates that it thinks the filename for the swift-testing build product is the XCTest bundle's executable. This PR plumbs through the two build parameters arguments to everywhere in `swift test` and `swift build` that needs them and resolves the issue.

After #7353 landed, I noticed that the build products for test targets were not being emitted correctly. swift-testing and XCTest produce separate build products (with distinct names) but this wasn't happening as intended. It turns out that the changes to split `buildParameters` into `productsBuildParameters` and `toolsBuildParameters` weren't fully propagated to our testing infrastructure. I also noticed `SWIFT_PM_SUPPORTS_SWIFT_TESTING` wasn't being set correctly anymore (same root cause) although we've decided to ignore that flag over in swift-testing anyway (see swiftlang/swift-testing#376.) This regression caused build failures in swift-testing (e.g. [here](https://ci.swift.org/job/pr-swift-testing-macos/663/console)) with the telltale failure signature: > /Users/ec2-user/jenkins/workspace/pr-swift-testing-macos/branch-main/swift-testing/.build/x86_64-apple-macosx/debug/swift-testingPackageTests.xctest/Contents/MacOS/swift-testingPackageTests: /Users/ec2-user/jenkins/workspace/pr-swift-testing-macos/branch-main/swift-testing/.build/x86_64-apple-macosx/debug/swift-testingPackageTests.xctest/Contents/MacOS/swift-testingPackageTests: cannot execute binary file Which indicates that it thinks the filename for the swift-testing build product is the XCTest bundle's executable. This PR plumbs through the two build parameters arguments to everywhere in `swift test` and `swift build` that needs them and resolves the issue. --------- Co-authored-by: Max Desiatov <m_desiatov@apple.com>

…swiftlang#7352)" (swiftlang#7353) Reverts swiftlang#7352. Modified to build tests for the host triple when any macros are added to test modules directly as dependencies.

Benchmarking modules graph is now generalized with `syntheticModulesGraph` function. It also uncovered a bug in the previous `SyntheticModulesGraph`, which didn't pass generated `TargetDescription`s array to `loadModulesGraph`, which is fixed now. New `SyntheticModulesGraphWithMacros` calls `syntheticModulesGraph` with `includeMacros: true` argument, which splits all modules in three parts: library modules, macros modules that library modules depend on, and macro dependencies that macros depend on. This allows us to track potential performance regressions in swiftlang#7353.

…#7508) After swiftlang#7353 landed, I noticed that the build products for test targets were not being emitted correctly. swift-testing and XCTest produce separate build products (with distinct names) but this wasn't happening as intended. It turns out that the changes to split `buildParameters` into `productsBuildParameters` and `toolsBuildParameters` weren't fully propagated to our testing infrastructure. I also noticed `SWIFT_PM_SUPPORTS_SWIFT_TESTING` wasn't being set correctly anymore (same root cause) although we've decided to ignore that flag over in swift-testing anyway (see swiftlang/swift-testing#376.) This regression caused build failures in swift-testing (e.g. [here](https://ci.swift.org/job/pr-swift-testing-macos/663/console)) with the telltale failure signature: > /Users/ec2-user/jenkins/workspace/pr-swift-testing-macos/branch-main/swift-testing/.build/x86_64-apple-macosx/debug/swift-testingPackageTests.xctest/Contents/MacOS/swift-testingPackageTests: /Users/ec2-user/jenkins/workspace/pr-swift-testing-macos/branch-main/swift-testing/.build/x86_64-apple-macosx/debug/swift-testingPackageTests.xctest/Contents/MacOS/swift-testingPackageTests: cannot execute binary file Which indicates that it thinks the filename for the swift-testing build product is the XCTest bundle's executable. This PR plumbs through the two build parameters arguments to everywhere in `swift test` and `swift build` that needs them and resolves the issue. --------- Co-authored-by: Max Desiatov <m_desiatov@apple.com>

After #7353 landed, I noticed that the build products for test targets were not being emitted correctly. swift-testing and XCTest produce separate build products (with distinct names) but this wasn't happening as intended. It turns out that the changes to split `buildParameters` into `productsBuildParameters` and `toolsBuildParameters` weren't fully propagated to our testing infrastructure. I also noticed `SWIFT_PM_SUPPORTS_SWIFT_TESTING` wasn't being set correctly anymore (same root cause) although we've decided to ignore that flag over in swift-testing anyway (see swiftlang/swift-testing#376.) This regression caused build failures in swift-testing (e.g. [here](https://ci.swift.org/job/pr-swift-testing-macos/663/console)) with the telltale failure signature: > /Users/ec2-user/jenkins/workspace/pr-swift-testing-macos/branch-main/swift-testing/.build/x86_64-apple-macosx/debug/swift-testingPackageTests.xctest/Contents/MacOS/swift-testingPackageTests: /Users/ec2-user/jenkins/workspace/pr-swift-testing-macos/branch-main/swift-testing/.build/x86_64-apple-macosx/debug/swift-testingPackageTests.xctest/Contents/MacOS/swift-testingPackageTests: cannot execute binary file Which indicates that it thinks the filename for the swift-testing build product is the XCTest bundle's executable. This PR plumbs through the two build parameters arguments to everywhere in `swift test` and `swift build` that needs them and resolves the issue. --------- Co-authored-by: Max Desiatov <m_desiatov@apple.com>

…swiftlang#7352)" (swiftlang#7353) Reverts swiftlang#7352. Modified to build tests for the host triple when any macros are added to test modules directly as dependencies.

Benchmarking modules graph is now generalized with `syntheticModulesGraph` function. It also uncovered a bug in the previous `SyntheticModulesGraph`, which didn't pass generated `TargetDescription`s array to `loadModulesGraph`, which is fixed now. New `SyntheticModulesGraphWithMacros` calls `syntheticModulesGraph` with `includeMacros: true` argument, which splits all modules in three parts: library modules, macros modules that library modules depend on, and macro dependencies that macros depend on. This allows us to track potential performance regressions in swiftlang#7353.

…#7508) After swiftlang#7353 landed, I noticed that the build products for test targets were not being emitted correctly. swift-testing and XCTest produce separate build products (with distinct names) but this wasn't happening as intended. It turns out that the changes to split `buildParameters` into `productsBuildParameters` and `toolsBuildParameters` weren't fully propagated to our testing infrastructure. I also noticed `SWIFT_PM_SUPPORTS_SWIFT_TESTING` wasn't being set correctly anymore (same root cause) although we've decided to ignore that flag over in swift-testing anyway (see swiftlang/swift-testing#376.) This regression caused build failures in swift-testing (e.g. [here](https://ci.swift.org/job/pr-swift-testing-macos/663/console)) with the telltale failure signature: > /Users/ec2-user/jenkins/workspace/pr-swift-testing-macos/branch-main/swift-testing/.build/x86_64-apple-macosx/debug/swift-testingPackageTests.xctest/Contents/MacOS/swift-testingPackageTests: /Users/ec2-user/jenkins/workspace/pr-swift-testing-macos/branch-main/swift-testing/.build/x86_64-apple-macosx/debug/swift-testingPackageTests.xctest/Contents/MacOS/swift-testingPackageTests: cannot execute binary file Which indicates that it thinks the filename for the swift-testing build product is the XCTest bundle's executable. This PR plumbs through the two build parameters arguments to everywhere in `swift test` and `swift build` that needs them and resolves the issue. --------- Co-authored-by: Max Desiatov <m_desiatov@apple.com>

…#7353) Reverts #7352. Modified to build tests for the host triple when any macros are added to test modules directly as dependencies. (cherry picked from commit cb3b085) # Conflicts: # CHANGELOG.md # Sources/Build/BuildDescription/SwiftTargetBuildDescription.swift # Sources/Build/BuildManifest/LLBuildManifestBuilder.swift # Sources/Build/BuildPlan/BuildPlan.swift # Sources/Commands/SwiftTestCommand.swift # Sources/Commands/Utilities/PluginDelegate.swift # Sources/Commands/Utilities/TestingSupport.swift # Sources/PackageGraph/ModulesGraph+Loading.swift # Sources/PackageGraph/ModulesGraph.swift # Sources/SPMTestSupport/MockBuildTestHelper.swift # Sources/SPMTestSupport/MockPackageGraphs.swift # Sources/SPMTestSupport/PackageGraphTester.swift # Sources/SPMTestSupport/ResolvedTarget+Mock.swift # Tests/BuildTests/ClangTargetBuildDescriptionTests.swift # Tests/BuildTests/CrossCompilationBuildPlanTests.swift # Tests/SourceKitLSPAPITests/SourceKitLSPAPITests.swift

After #7353 landed, I noticed that the build products for test targets were not being emitted correctly. swift-testing and XCTest produce separate build products (with distinct names) but this wasn't happening as intended. It turns out that the changes to split `buildParameters` into `productsBuildParameters` and `toolsBuildParameters` weren't fully propagated to our testing infrastructure. I also noticed `SWIFT_PM_SUPPORTS_SWIFT_TESTING` wasn't being set correctly anymore (same root cause) although we've decided to ignore that flag over in swift-testing anyway (see swiftlang/swift-testing#376.) This regression caused build failures in swift-testing (e.g. [here](https://ci.swift.org/job/pr-swift-testing-macos/663/console)) with the telltale failure signature: > /Users/ec2-user/jenkins/workspace/pr-swift-testing-macos/branch-main/swift-testing/.build/x86_64-apple-macosx/debug/swift-testingPackageTests.xctest/Contents/MacOS/swift-testingPackageTests: /Users/ec2-user/jenkins/workspace/pr-swift-testing-macos/branch-main/swift-testing/.build/x86_64-apple-macosx/debug/swift-testingPackageTests.xctest/Contents/MacOS/swift-testingPackageTests: cannot execute binary file Which indicates that it thinks the filename for the swift-testing build product is the XCTest bundle's executable. This PR plumbs through the two build parameters arguments to everywhere in `swift test` and `swift build` that needs them and resolves the issue. --------- Co-authored-by: Max Desiatov <m_desiatov@apple.com> (cherry picked from commit 5a4c024) # Conflicts: # Sources/Commands/SwiftBuildCommand.swift # Sources/Commands/SwiftTestCommand.swift # Sources/Commands/Utilities/TestingSupport.swift

@bnbarham

**Explanation**: Macros cross-compiled by SwiftPM with Swift SDKs should be correctly built, loaded, and evaluated for the host triple. **Scope**: isolated to modules dependency resolution and llbuild, does not impact code related to XCBuild. **Risk**: medium, known issues were addressed on `main` and are cherry-picked here, with no new issues reported for a few weeks now. **Testing**: added unit tests, manual end-to-end testing done with existing Swift SDKs. **Issue**: rdar://105991372 **Reviewers**: @bnbarham @xedin @neonichu (cherry picked from commit cb3b085, #7353) ``` # Conflicts: # CHANGELOG.md # Sources/Build/BuildDescription/SwiftTargetBuildDescription.swift # Sources/Build/BuildManifest/LLBuildManifestBuilder.swift # Sources/Build/BuildPlan/BuildPlan.swift # Sources/Commands/SwiftTestCommand.swift # Sources/Commands/Utilities/PluginDelegate.swift # Sources/Commands/Utilities/TestingSupport.swift # Sources/PackageGraph/ModulesGraph+Loading.swift # Sources/PackageGraph/ModulesGraph.swift # Sources/SPMTestSupport/MockBuildTestHelper.swift # Sources/SPMTestSupport/MockPackageGraphs.swift # Sources/SPMTestSupport/PackageGraphTester.swift # Sources/SPMTestSupport/ResolvedTarget+Mock.swift # Tests/BuildTests/ClangTargetBuildDescriptionTests.swift # Tests/BuildTests/CrossCompilationBuildPlanTests.swift # Tests/SourceKitLSPAPITests/SourceKitLSPAPITests.swift ``` (cherry picked from commit b9eb3c1, #7493) ``` # Conflicts: # Sources/PackageGraph/ModulesGraph+Loading.swift # Sources/PackageGraph/Resolution/ResolvedPackage.swift ``` (cherry picked from commit 5a4c024, #7508) ``` # Conflicts: # Sources/Commands/SwiftBuildCommand.swift # Sources/Commands/SwiftTestCommand.swift # Sources/Commands/Utilities/TestingSupport.swift ``` (cherry picked from commit 8db2401, #7519) ``` # Conflicts: # Tests/BuildTests/CrossCompilationBuildPlanTests.swift ``` --------- Co-authored-by: Jonathan Grynspan <jgrynspan@apple.com> Co-authored-by: Ben Barham <ben_barham@apple.com> Co-authored-by: Yuta Saito <kateinoigakukun@gmail.com> Co-authored-by: Pavel Yaskevich <xedin@apache.org>

Revert "Revert "Support macros when cross-compiling (#7118)" (#7352)"

be939ca

This reverts commit adfbd9a.

This was referenced Feb 20, 2024

[DNM] cross-repo testing PR swiftlang/swift#65627

Closed

[DNM] cross-repo testing PR swiftlang/swift-syntax#2504

Draft

MaxDesiatov self-assigned this Feb 20, 2024

MaxDesiatov added the bug label Feb 20, 2024

MaxDesiatov added the cross-compilation label Feb 25, 2024

MaxDesiatov added 2 commits March 8, 2024 11:57

Fix build issues after conflicts resolution

14482e0

MaxDesiatov force-pushed the revert-7352-maxd/revert-cross-compilation branch from 3b0105b to 14482e0 Compare March 8, 2024 12:13

Fix SourceKitLSPAPITests

47dd496

Make change conditional on --experimental-macros-cross-compilation

10c7cd1

MaxDesiatov changed the title ~~Revert "Revert "Support macros when cross-compiling (#7118)""~~ Introduce --experimental-macros-cross-compilation flag Mar 8, 2024

MaxDesiatov added 2 commits March 20, 2024 15:00

Merge branch 'main' of github.com:apple/swift-package-manager into re…

698440d

…vert-7352-maxd/revert-cross-compilation

Revert "Make change conditional on `--experimental-macros-cross-compi…

d5c9831

…lation`" This reverts commit 10c7cd1.

Add tests for macros testing modules graph per the status quo

1a2f110

Add a trivial graph x-compilation test, fix plugins test

aae5a2c

Fix testMacrosTests

244b4cc

MaxDesiatov changed the title ~~Introduce --experimental-macros-cross-compilation flag~~ Revert "Revert "Support macros when cross-compiling (#7118)" (#7352)" Mar 25, 2024

xedin reviewed Mar 25, 2024

View reviewed changes

Sources/Build/BuildDescription/ResolvedTarget+BuildDescription.swift Outdated Show resolved Hide resolved

finagolfin mentioned this pull request Mar 27, 2024

Cross-compiling packages which contain macros doesn't work #6950

Closed

MaxDesiatov mentioned this pull request Apr 17, 2024

Add synthetic modules-with-macros graph benchmarks #7466

Merged

bnbarham requested a review from neonichu April 17, 2024 17:44

MaxDesiatov enabled auto-merge (squash) April 17, 2024 17:48

MaxDesiatov removed the DO NOT MERGE label Apr 17, 2024

MaxDesiatov merged commit cb3b085 into main Apr 17, 2024
5 checks passed

MaxDesiatov deleted the revert-7352-maxd/revert-cross-compilation branch April 17, 2024 20:27

kabiroberai mentioned this pull request Apr 23, 2024

How to compile a .node file that supports both arm64 and x86 architectures simultaneously? kabiroberai/node-swift#19

Closed

stmontgomery mentioned this pull request Apr 23, 2024

swift-testing does not build with ToT SwiftPM #7479

Closed

grynspan mentioned this pull request Apr 26, 2024

Fix test targets that depend on macros and swift-testing #7508

Merged

MaxDesiatov restored the revert-7352-maxd/revert-cross-compilation branch June 6, 2024 09:58

MaxDesiatov deleted the revert-7352-maxd/revert-cross-compilation branch June 6, 2024 10:00

MaxDesiatov mentioned this pull request Jun 6, 2024

[6.0] Support macros when cross-compiling #7640

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Revert "Revert "Support macros when cross-compiling (#7118)" (#7352)" #7353

Revert "Revert "Support macros when cross-compiling (#7118)" (#7352)" #7353

MaxDesiatov commented Feb 20, 2024 •

edited

Loading

MaxDesiatov commented Feb 20, 2024

MaxDesiatov commented Feb 21, 2024

MaxDesiatov commented Feb 26, 2024

MaxDesiatov commented Feb 26, 2024

MaxDesiatov commented Mar 8, 2024

MaxDesiatov commented Mar 20, 2024

MaxDesiatov commented Mar 20, 2024

MaxDesiatov commented Mar 21, 2024

MaxDesiatov commented Mar 25, 2024

grynspan commented Mar 31, 2024

MaxDesiatov commented Apr 16, 2024

MaxDesiatov commented Apr 17, 2024

rauhul commented Apr 17, 2024

finagolfin commented Apr 25, 2024

Revert "Revert "Support macros when cross-compiling (#7118)" (#7352)" #7353

Revert "Revert "Support macros when cross-compiling (#7118)" (#7352)" #7353

Conversation

MaxDesiatov commented Feb 20, 2024 • edited Loading

MaxDesiatov commented Feb 20, 2024

MaxDesiatov commented Feb 21, 2024

MaxDesiatov commented Feb 26, 2024

MaxDesiatov commented Feb 26, 2024

MaxDesiatov commented Mar 8, 2024

MaxDesiatov commented Mar 20, 2024

MaxDesiatov commented Mar 20, 2024

MaxDesiatov commented Mar 21, 2024

MaxDesiatov commented Mar 25, 2024

grynspan commented Mar 31, 2024

MaxDesiatov commented Apr 16, 2024

MaxDesiatov commented Apr 17, 2024

rauhul commented Apr 17, 2024

finagolfin commented Apr 25, 2024

MaxDesiatov commented Feb 20, 2024 •

edited

Loading