Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Just autolink everything #16349

Merged
merged 7 commits into from
May 4, 2018

Conversation

jrose-apple
Copy link
Contributor

@jrose-apple jrose-apple commented May 3, 2018

Yet another attempt at fixing rdar://problem/39338239.

The problem: If module A uses some inlinable code from module B (including function default arguments), and that code uses a symbol in module C, but module A doesn't itself import module C, we'll get a link error.

Attempt 1: Record which imports are used within inlinable code (#16326).
Issue 1: Module B might not have imported module C directly (comment).

Attempt 2: Record which modules are used within inlinable code by adding them to the list of imports (#16299).
Issue 2: This affects name lookup. Declarations in module C might be shadowed by whatever imported module C.

Attempt 3: Record a separate list of modules used within inlinable code, for autolinking purposes only (#16317).
Issue 3: Doesn't play nice with Apple-internal SDK tricks using the newly-introduced export_as autolinking feature (apple/swift-clang#124).

Attempt 4 (this PR): Just autolink everything that's imported, publicly or privately. This isn't as bad as it sounds; we already drag in private imports just to load a swiftmodule, and C modules tend to re-export everything all the time anyway for compatibility with textual includes. We'll come up with something better post-4.2. (This means reverting a number of Slava's changes in pursuit of the earlier attempts.)

Rejected alternative: use Attempt 3's model, but save autolinking info directly, instead of saving a list of modules. Rejected because really we're just making this more complex when we need a safe fix for 4.2.

@jrose-apple
Copy link
Contributor Author

@swift-ci Please test

@jrose-apple
Copy link
Contributor Author

@swift-ci Please test source compatibility

Copy link
Contributor

@graydon graydon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming it's safe to yank @_usableFromInline (nobody else is using it? we made no promises about it?) this seems reasonable & conservative.

(I also have no idea what sort of corner this paints us into post-4.2 but I do not feel like I have enough of the full picture to know what you are alluding to as a "real fix" later, or whether this makes it harder, etc. I.e. I feel my opinion on the local change doesn't necessarily extend to an opinion on the overall strategy)

@swift-ci
Copy link
Contributor

swift-ci commented May 3, 2018

Build failed
Swift Test Linux Platform
Git Sha - f6c21cc600b3b1404a814bef996becc46c0889fa

@jrose-apple
Copy link
Contributor Author

Linux test failure is unrelated

@swift-ci Please test Linux

@jrose-apple
Copy link
Contributor Author

Assuming it's safe to yank @_usableFromInline (nobody else is using it? we made no promises about it?) this seems reasonable & conservative.

Yep, Slava added it within the last two weeks, and only for this purpose.

(I also have no idea what sort of corner this paints us into post-4.2 but I do not feel like I have enough of the full picture to know what you are alluding to as a "real fix" later, or whether this makes it harder, etc. I.e. I feel my opinion on the local change doesn't necessarily extend to an opinion on the overall strategy)

At worst, we'll have to keep this logic around for Swift 4 mode. But in practice I don't think that this will affect anything outside of inlinable code—everywhere you could directly reference a name, autolinking already picked it up.

You can see some of my thoughts on where to go for real on the forums.

@swift-ci
Copy link
Contributor

swift-ci commented May 3, 2018

Build failed
Swift Test OS X Platform
Git Sha - f6c21cc600b3b1404a814bef996becc46c0889fa

@graydon
Copy link
Contributor

graydon commented May 3, 2018

You can see some of my thoughts on where to go for real on the forums.

Makes sense, thanks for clarifying!

@swift-ci
Copy link
Contributor

swift-ci commented May 3, 2018

Build failed
Swift Test Linux Platform
Git Sha - f6c21cc600b3b1404a814bef996becc46c0889fa

Copy link
Member

@DougGregor DougGregor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, this looks like the safest approach. Thank you, Jordan.

@jrose-apple jrose-apple force-pushed the just-autolink-everything branch from f6c21cc to f51e027 Compare May 3, 2018 23:20
@jrose-apple
Copy link
Contributor Author

Source compat tests still going at https://ci.swift.org/job/swift-PR-source-compat-suite/1152/. The only changes here are test updates.

@swift-ci Please test

@swift-ci
Copy link
Contributor

swift-ci commented May 3, 2018

Build failed
Swift Test Linux Platform
Git Sha - f6c21cc600b3b1404a814bef996becc46c0889fa

@swift-ci
Copy link
Contributor

swift-ci commented May 3, 2018

Build failed
Swift Test OS X Platform
Git Sha - f6c21cc600b3b1404a814bef996becc46c0889fa

@jrose-apple
Copy link
Contributor Author

Source compat tests passed, now just waiting for regular tests.

@slavapestov
Copy link
Contributor

Why are you reverting the cleanups? I don't think they're actually related to the solution and made the code a bit simpler.

@jrose-apple
Copy link
Contributor Author

jrose-apple commented May 4, 2018

I needed forAllImportedModules back, and I wanted to keep the diff from the starting point smaller. I also don't think removing the void-returning overloads makes things better.

@jrose-apple jrose-apple force-pushed the just-autolink-everything branch from f51e027 to c9ad24e Compare May 4, 2018 01:16
@jrose-apple
Copy link
Contributor Author

Full Apple platform tests continue here: https://ci.swift.org/job/swift-PR-osx/4726/

@swift-ci Please test Linux

@jrose-apple
Copy link
Contributor Author

@swift-ci Please smoke test macOS

@swift-ci
Copy link
Contributor

swift-ci commented May 4, 2018

Build failed
Swift Test Linux Platform
Git Sha - f51e02767fe91d34679703efb566d523a13381a3

@jrose-apple
Copy link
Contributor Author

I just can't win on this one. @compnerd, how are these tests supposed to work on non-Apple platforms?

…nces from inlinable functions (swiftlang#16326)"

This reverts commit ee6e190. It's not
sufficient to solve the problem, and the choices were to do something
more complicated, or just take a simple brute force approach. We're
going with the latter.
…able-vs-autolinking"

This reverts commit bb16ee0,
reversing changes made to a8d831f.
It's not sufficient to solve the problem, and the choices were to do
something more complicated, or just take a simple brute force
approach. We're going with the latter.
And disable them on non-ObjC platforms again, because those overlays
depend on a stdlib that contains _ObjectiveCBridgeable.
This is unfortunate in that it makes the linker do extra work, but in
practice it probably doesn't matter much, and meanwhile it handles all
our problems with @inlinable.

Alternate solution to rdar://problem/39338239
One bit of Slava's cleanup that I /can/ keep. No one is using this
parameter anymore.
@jrose-apple jrose-apple force-pushed the just-autolink-everything branch from c9ad24e to fb59305 Compare May 4, 2018 04:58
@jrose-apple
Copy link
Contributor Author

@swift-ci Please test Linux

@jrose-apple
Copy link
Contributor Author

@swift-ci Please smoke test macOS

@swift-ci
Copy link
Contributor

swift-ci commented May 4, 2018

Build failed
Swift Test Linux Platform
Git Sha - c9ad24ed3d6da06277d2a22ad6e4efe363b29a95

@compnerd
Copy link
Member

compnerd commented May 4, 2018

@jrose-apple the IRGen tests should definitely work on all targets, nothing there is an execution test, so we should be able to enable them on all targets. ObjC interop CodeGen is entirely controlled at the driver level. When I ran the tests _ObjectiveCBridgeable didn't seem to be needed on any of them (and is also reflected in the fact that the tests were passing on the buildbots). It seems unfortunate to have to disable them for now. However, @millenomi is working on enabling _ObjectiveCBridgeable on Linux as well, so we should be able to re-enable it if that is causing issues right now.

@jrose-apple
Copy link
Contributor Author

The tests in question were using the mock SDK but not the mock overlays, which isn't exactly a useful configuration. But yeah, hopefully Lily's work will get us further.

@jrose-apple
Copy link
Contributor Author

@swift-ci Please test Linux

@swift-ci
Copy link
Contributor

swift-ci commented May 4, 2018

Build failed
Swift Test Linux Platform
Git Sha - fb59305

@jrose-apple
Copy link
Contributor Author

@swift-ci Please test Linux

@jrose-apple jrose-apple merged commit 76ef276 into swiftlang:master May 4, 2018
@jrose-apple jrose-apple deleted the just-autolink-everything branch May 4, 2018 19:55
@davezarzycki
Copy link
Contributor

The breaks Red Hat Fedora 28:

FAILED: lib/swift/linux/x86_64/libswiftStdlibUnicodeUnittest.so 
: && /usr/local/bin/clang++ -fPIC -Wno-c11-extensions -Wno-gnu-anonymous-struct -Wno-unused-local-typedef -Wno-dollar-in-identifier-extension -Wno-unused-private-field -Wno-gnu-statement-expression -Wno-c99-extensions -Wno-nullability-extension -Wno-format-pedantic -Wno-global-constructors -Wno-four-char-constants -Wno-c++17-extensions -Wno-global-constructors -Werror=macro-redefined -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fcolor-diagnostics -ffunction-sections -fdata-sections -Wno-nested-anon-types -Werror=switch -Wdocumentation -Wimplicit-fallthrough -Wunreachable-code -Woverloaded-virtual -DOBJC_OLD_DISPATCH_PROTOTYPES=0 -DLLVM_DISABLE_ABI_BREAKING_CHECKS_ENFORCING=1 -DSWIFT_RUNTIME_ENABLE_GUARANTEED_NORMAL_ARGUMENTS=1 -O2  -stdlib=libc++ -Wl,-z,defs -Wl,-z,nodelete -fuse-ld=lld -Wl,--color-diagnostics   -target x86_64-unknown-linux-gnu -lpthread -latomic -ldl -fuse-ld=lld  "-L/home/dave/s/u/t/./lib/swift/linux/x86_64" "-L/home/dave/s/u/t/./bin/../lib/swift/linux/x86_64" "-L/home/dave/s/u/t/./bin/../lib/swift/linux" "-L/usr/lib64" "-L/usr/lib64" -shared -Wl,-soname,libswiftStdlibUnicodeUnittest.so -o lib/swift/linux/x86_64/libswiftStdlibUnicodeUnittest.so tools/swift/stdlib/private/StdlibUnicodeUnittest/linux/x86_64/StdlibUnicodeUnittest.o lib/swift/linux/x86_64/swiftrt.o  -Wl,-rpath,"\$ORIGIN:/usr/lib/swift/linux" lib/swift/linux/x86_64/libswiftStdlibUnittest.so lib/swift/linux/x86_64/libswiftCore.so && :
/usr/local/bin/ld.lld: error: undefined symbol: _swift_FORCE_LOAD_$_swiftGlibc
>>> referenced by StdlibUnicodeUnittest.o
>>>               tools/swift/stdlib/private/StdlibUnicodeUnittest/linux/x86_64/StdlibUnicodeUnittest.o:($S21StdlibUnicodeUnittest7UTFTestV5FlagsVN)
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
[7+5] Building CXX object tools/swift/unittests/runtime/CMakeFiles/SwiftRuntimeTests.dir/Array.cpp.o
FAILED: lib/swift/linux/x86_64/libswiftStdlibCollectionUnittest.so 
: && /usr/local/bin/clang++ -fPIC -Wno-c11-extensions -Wno-gnu-anonymous-struct -Wno-unused-local-typedef -Wno-dollar-in-identifier-extension -Wno-unused-private-field -Wno-gnu-statement-expression -Wno-c99-extensions -Wno-nullability-extension -Wno-format-pedantic -Wno-global-constructors -Wno-four-char-constants -Wno-c++17-extensions -Wno-global-constructors -Werror=macro-redefined -stdlib=libc++ -fPIC -fvisibility-inlines-hidden -Werror=date-time -Werror=unguarded-availability-new -std=c++11 -Wall -W -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic -Wno-long-long -Wcovered-switch-default -Wnon-virtual-dtor -Wdelete-non-virtual-dtor -Wstring-conversion -fcolor-diagnostics -ffunction-sections -fdata-sections -Wno-nested-anon-types -Werror=switch -Wdocumentation -Wimplicit-fallthrough -Wunreachable-code -Woverloaded-virtual -DOBJC_OLD_DISPATCH_PROTOTYPES=0 -DLLVM_DISABLE_ABI_BREAKING_CHECKS_ENFORCING=1 -DSWIFT_RUNTIME_ENABLE_GUARANTEED_NORMAL_ARGUMENTS=1 -O2  -stdlib=libc++ -Wl,-z,defs -Wl,-z,nodelete -fuse-ld=lld -Wl,--color-diagnostics   -target x86_64-unknown-linux-gnu -lpthread -latomic -ldl -fuse-ld=lld  "-L/home/dave/s/u/t/./lib/swift/linux/x86_64" "-L/home/dave/s/u/t/./bin/../lib/swift/linux/x86_64" "-L/home/dave/s/u/t/./bin/../lib/swift/linux" "-L/usr/lib64" "-L/usr/lib64" -shared -Wl,-soname,libswiftStdlibCollectionUnittest.so -o lib/swift/linux/x86_64/libswiftStdlibCollectionUnittest.so tools/swift/stdlib/private/StdlibCollectionUnittest/linux/x86_64/StdlibCollectionUnittest.o lib/swift/linux/x86_64/swiftrt.o  -Wl,-rpath,"\$ORIGIN:/usr/lib/swift/linux" lib/swift/linux/x86_64/libswiftStdlibUnittest.so lib/swift/linux/x86_64/libswiftCore.so && :
/usr/local/bin/ld.lld: error: undefined symbol: _swift_FORCE_LOAD_$_swiftGlibc
>>> referenced by StdlibCollectionUnittest.o
>>>               tools/swift/stdlib/private/StdlibCollectionUnittest/linux/x86_64/StdlibCollectionUnittest.o:(_swift_FORCE_LOAD_$_swiftGlibc_$_StdlibCollectionUnittest)
clang-7: error: linker command failed with exit code 1 (use -v to see invocation)
[1+5] Building CXX object tools/swift/unittests/runtime/CMakeFiles/SwiftRuntimeTests.dir/Metadata.cpp.o
ninja: build stopped: subcommand failed.

@jrose-apple
Copy link
Contributor Author

Those seem like reasonable errors, actually. Are we missing a build dependency on swiftGlibc?

@jrose-apple
Copy link
Contributor Author

Hm, there's a SWIFT_MODULE_DEPENDS_LINUX Glibc in the dependencies for swiftStdlibUnittest, so probably things are okay there…

…oh, right, we don't use autolinking in our CMake build, but we will still pick up the FORCE_LOAD symbols. I wonder why this works at all on Ubuntu?

@davezarzycki
Copy link
Contributor

I don't know. I use unified builds if it matters. Also, this pull request causes a 46.5% performance regression on my box when building Swift.o.

@jrose-apple
Copy link
Contributor Author

…that makes no sense. Swift.o only imports one thing. It's pretty much not possible for that to be this PR.

@davezarzycki
Copy link
Contributor

Hmm… I think I misread where your branch started. Let me double check

@davezarzycki
Copy link
Contributor

Hi @jrose-apple – Just to circle back, the performance problem I observed is not the fault of this pull request. That being said, the performance problem I'm observing is truly weird. I'll post to the forum.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants