Add support to iOS / Mac ARM64 #9699

EduardoRFS · 2020-06-22T00:14:20Z

Requirements

Apple / Darwin ARM64 assembler:

global label should be prefixed with _
local label should be prefixed with L
use assembly macros instead of C macros
remove .type and .size

Runtime requirements:

stop using x18 as the kernel sets it to 0 at anytime
only x0 to x9 are saved between calls when the linker stub is added

About this PR

This PR was based on the #1084 but it diverges on some details and it also addresses some problems, namely, shared libraries on iOS and ./configure support.

~~I tried to keep the ABI as close as possible to Linux ARM64, they're identical if we keep x15 as the ADDITIONAL_ARG and when the OCaml code is in the main binary.~~
After some changes, now both ARM64 platforms uses x8 as ADDITIONAL_ARG, so the only remaining difference in the ABI is #9752 and that we do only x0-x7 for ARGS.

But for iOS if the OCaml code is in a shared library it mess with every register except x0-x9 when making function calls, even to symbols in the same shared library. The ABI is x0-x7 for ARGS, x8 seems to be used for syscalls so it is also saved and x9 is saved in the same package as x8.

And to support iOS in a shared library this ABI needs to be followed, so x0-x7 was used for ARGS and x8 was used as the additional ARG.

Changes to Linux ARM64

It is a requirement to not use x18, by moving some registers it was possible to stop using it without any visible(to me) performance impact as the data dependency and the instructions was kept 1:1.

But as x18 is still safe to use, and x15 is also used as a temporary register, it seems like a good idea to use x18 as the ADDITIONAL ARG instead of x15.

That can easily be reverted if it isn't desired.

Current state + Tests

Currently I can compile it using a cross compiler and inside an iPhone with jailbreak is also possible to run the testsuite if you patch the iPhoneOS.sdk to support system(). Currently the best way to run the tests is exactly like that, an iPhone up to iPhone X with jailbreak on an ssh, but maybe Apple will release a Mac ARM.

The test suite is almost entirely green, I'm looking on how to solve all of them, but overall the support is great

https://gist.githubusercontent.com/EduardoRFS/d067cff0aebec4df0fe55539669fff05/raw/10e071ba62a094c37e2f140835ed33a2cacc3211/test-result-ocaml+ios.txt

Known causes:

statmemprof/internal.ml fails because iOS supports only 1008 on the stack size, so it doesn't fit in the stack

EduardoRFS · 2020-06-22T00:18:47Z

as you may be interested: @shindere

runtime/sys.c

shindere · 2020-06-22T15:54:46Z

EduardoRFS (2020/06/21 17:18 -0700):

as you may be interested: @shindere

Yes yes, I'm following. Just that I have too little time to deal with it at the moment, but it's on my lists.

EduardoRFS · 2020-06-23T01:39:55Z

Also backported to 4.10, same state as the trunk

https://github.com/EduardoRFS/ocaml/tree/4.10+ios

xavierleroy · 2020-06-24T16:37:41Z

I haven't fully reviewed yet, and hope to do it soon, but this looks very good, and extremely timely! Thank you @EduardoRFS .

It would be very nice to add references to the iOS ABI (and the soon-to-be-revealed macOS one too!) in file asmcomp/arm64/NOTES.md. Is this the reference document you worked from? https://developer.apple.com/library/archive/documentation/Xcode/Conceptual/iPhoneOSABIReference/Articles/ARM64FunctionCallingConventions.html

EduardoRFS · 2020-06-24T18:49:03Z

Added that, but it's not the full story, like x8 and x9 being safe across calls, it's in the linker code, as you can see the stub mentioning it. Couldn't found any reference for that as most details of that. Maybe mentioning that in the code?

dyld_stub_binder

m-schmidt · 2020-06-29T13:19:47Z

Three remarks to this pull request:

the target triple for the new ARM based Macs seems to be not aarch64-*-darwin* but e.g. arm-apple-darmin20.0.0
with a patched target triple make world.opt works
make tests reports 4 failures:

List of failed tests:
tests/unwind/'driver.ml' with 1.1.1.1.1.1 (run)
tests/unboxed-primitive-args/'test.ml' with 1.1.1.1.1.1 (check-program-output)
tests/link-test/'test.ml' with 2.1.1.1.1.1.1.1.1.1.1.1.1 (check-ocamlopt.byte-output)
tests/required-external/'main.ml' with 2.1.1.1.1 (check-ocamlopt.byte-output)

Edit: attached log files of testsuite: log.txt

xavierleroy · 2020-06-29T15:14:22Z

@m-schmidt: thanks a lot for the early testing, the results are extremely encouraging!

tests/unwind is a macos-specific thing for Spacetime profiling; possibly Spacetime needs to be turned off on macOS/ARM64.
tests/unboxed-primitive-args may point to a mismatch in calling conventions.
tests/link-test and tests/required-external both complain about a .o being put in a .a and having size <> 0 mod 8; should be easy to track down.

xavierleroy

Commit fdcfe0d must be removed before merging because it breaks the bootstrap. I understand why it is useful in a cross-compilation setting, but cross-compilation will have to be handled differently and in separate future work.

xavierleroy · 2020-06-29T15:22:41Z

Makefile

@@ -1014,7 +1014,7 @@ toplevel/opttoploop.cmx: otherlibs/dynlink/dynlink.cmxa
 make_opcodes := tools/make_opcodes$(EXE)

 bytecomp/opcodes.ml: runtime/caml/instruct.h $(make_opcodes)
-	runtime/ocamlrun$(EXE) $(make_opcodes) -opcodes < $< > $@
+	$(CAMLRUN) $(make_opcodes) -opcodes < $< > $@


make_opcodes is generated by boot/ocamlc -use-prims runtime/primitives, so it must be run with runtime/camlrun$(EXE).

Will address it, thanks and also open a cross compilation PR.

If anyone is interested on why I made that, I got a native cross compiler to work(ocamlopt.opt), and this was a requirement.
https://github.com/EduardoRFS/reason-mobile/blob/master/patches/ocaml/files/make.cross.sh

xavierleroy

(I'm trying to review commit-by-commit and Github doesn't make this easy.)

Commit 2c84349: I agree the use of x18 was wrong. The original ARM64/Linux port did not use x18 at all.

ARG_DOMAIN_STATE_PTR could be any register used for parameter passing beyond r0...r3, which are used to pass up to 4 arguments to the OCaml code being called from C.

The proposed change is OK as long as we don't need a second temporary register in addition to x16. If we ever need a second temp, we'll go back to using r17 and could use r4, say, as ARG_DOMAIN_STATE_PTR.

xavierleroy

Commit 02dd612 is fine. Well spotted!

However the commit message should be made more informative:

"Avoid implicit conversion from integer to FP

This is flagged (rightly so) by clang -Wimplicit-int-float-conversion."

xavierleroy

Commit 0c718d3 is OK.
Commit cf8cc86 is OK.
Commit 7654d56: I'm not sure about it. The prudent thing to do would be to keep the .size and .type declarations for ARM64/Linux code.

runtime/arm64.S

xavierleroy

Commit 7db8f62 looks good to me.

Commit 51769c0 is OK, but it's a real pain that the macOS assembler doesn't support multiline C preprocessor macros like the GNU and LLVM assemblers.

Commit 7f6681b is misguided in my opinion. The compiled code should never use x18 for any purpose. But I'm pretty sure you can use x8 for the "ARG" register, see below why.

asmcomp/arm64/emit.mlp

asmcomp/arm64/proc.ml

runtime/arm64.S

xavierleroy

The last commit 2f1c175, looks good to me. A small suggestion below.

xavierleroy

This is the style comment I promised.

xavierleroy · 2020-06-29T16:02:16Z

asmcomp/arm64/emit.mlp

+let is_macosx =
+  Config.system = "macosx"


You could move this definition to asmcomp/arm64/arch.ml, so that you don't have to repeat it in selection.ml.

Also, the house style would omit is_ and call this variable macosx. For example in asmcomp/amd64/arch.ml, we have a win64 Boolean, not is_win64.

Also, the new brand name is macOS, so macOS or macos would work too.

I'm trying to keep it coherently with every other mention where it is macosx, if we want to rename, it should be in a future PR. But fully agreed on removing the is_ and moving to arch.ml

EduardoRFS · 2020-06-29T19:14:34Z

@xavierleroy thanks for the review, it's funny I had even implemented x8 instead of x18 for every platform, but I was thinking "it will slightly increase the spill rate, but if you agree on that, this is nice.

Also it will makes a lot more easier to check the overall code because we will not have any branching except by the calling convention.

Thx for the review, will address the points maded in the following days and fix the buggy tests.

EduardoRFS · 2020-06-29T19:18:29Z

@m-schmidt seriously, thanks, are you sure about the triplet? Like is that what clang -dumpmachine get's you? If so I can fix it. It was the same for iOS where the triplet is arm64-apple-ios13.5.0. So I was doing ./configure --host=aarch64-apple-darwin19.5.0 --build=aarch64-apple-darwin19.5.0.

Also if you can or know someone who can lend me an ssh I have time to address it, I'm doing this fulltime. And would love to address esy(and opam if needed).

m-schmidt · 2020-06-30T07:00:20Z

I get the following output:

$ clang -dumpmachine
arm64-apple-darwin20.0.0

But the configure script of OCaml prints:

configure: Configuring OCaml version 4.12.0+dev0-2020-04-22
checking build system type... arm-apple-darwin20.0.0
checking host system type... arm-apple-darwin20.0.0
checking target system type... arm-apple-darwin20.0.0

EduardoRFS · 2020-06-30T07:03:56Z

@m-schmidt thx, is exactly what I was expecting, the configure script doesn't know arm64 it's not an option, so I'm going to fix for both of them arm64-apple-darwin19.5.0 = Darwin ARM64. Which is the same problem as previously where x86_64-apple-darwin19.5.0 could be iOS x86_64 or mac x86_64.

Simple to fix, thanks for your help

xavierleroy · 2020-06-30T07:12:30Z

Configuration triplets are guessed by the config.guess script, which is straight from the FSF. Please try with the newest scripts first (https://savannah.gnu.org/projects/config), before modifying our configure.ac.

shindere · 2020-06-30T07:42:03Z

Naïve question: how do you test on iOS? Do you have to use a jailbroken hardware, or have you found another way? Thanks!

shindere · 2020-06-30T07:45:24Z

Xavier Leroy (2020/06/30 00:12 -0700):

Configuration triplets are guessed by the `config.guess` script, which is straight from the FSF. Please try with the newest scripts first (https://savannah.gnu.org/projects/config), before modifying our configure.ac.

Agreed, if the newer `config.guess` fixes the problem, we can replace our bundled one by this one. Thanks a lot for pointing that out, @xavierleroy

EduardoRFS · 2020-06-30T07:54:37Z

@shindere actually both, I use a jailbroken iPhone, quite simple to setup. But also I have an iOS emulator working (xnu-qemu-arm64), currently it doesn't support KVM so full software emulation, but I'm working on that.

shindere · 2020-06-30T08:05:47Z

Interesting, thanks! do you have to stick to a precise iOS version, or are you able to jailbreak again after even the most recent upgrades? I'd be interested in the emulator solution, too!

EduardoRFS · 2020-06-30T19:49:08Z

@shindere I can update my iPhone, every iPhone up to the iPhone X, has a permanent security problem into the ROM part of the bootloader and that leads to jailbreak. Look for checkra1n. So while iPhone X is supported that should still be valuable.

The emulator, you should be able to setup iOS 12 which is the same from a binary perspective as iOS 13. If you're using macOS following the tutorial on this README.md should work, for Linux you need some patches that I didn't publish to get the network to work. After having everything setup, it's just like any VM, with working SSH.

BTW I'm waiting the mac ARM, as it should be able to run almost exactly the same binaries as a normal iPhone when using the simulator. Except by some really low level features(SECURE_KERNEL).

xavierleroy · 2020-07-01T17:29:26Z

I had even implemented x8 instead of x18 for every platform, but I was thinking "it will slightly increase the spill rate, but if you agree on that, this is nice.

Right, I'm not worried at all about extra spilling here. For Iextcall that goes through caml_c_call, all registers are destroyed anyway. For Ialloc that goes through caml_allocN, we're not in "fast code" mode, instead we're optimizing for code size, so a tiny slowdown is OK.

avsm · 2020-07-27T21:21:00Z

I'm seeing lots of alignment warnings from ocamlopt generated code:

/Applications/Xcode-beta.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib:
archive member: src/state/opam_state.a(opamPackageVar.o) offset in archive not a
multiple of 8 (must be since member is an 64-bit object file)

Reading the manual page for ranlib reveals the worrying detail that the libtool(1) command (not to be confused with the GNU libtool) replaces ranlib now:

Libtool with -static is intended to replace ar(5) and ranlib

There's already a bit of a morass of workarounds for macOS-specific behaviour with ranlib/ar (especially around empty archives) in trunk, so I stopped looking there for the evening :-)

xavierleroy · 2020-07-28T07:52:47Z

Reading the manual page for ranlib reveals the worrying detail that the libtool(1) command (not to be confused with the GNU libtool) replaces ranlib now

I'm seeing the glass half-full: maybe libtool -static will be less finicky than ar? Time to investigate...

xavierleroy · 2020-07-28T13:58:10Z

I confirm everything works and builds without warnings if "ar" is configured to be a shell script that calls "libtool -shared" underneath. Nonetheless, it would be nice if a registered Apple developer could report the issue with "ar", as it is affecting other programs as well.

avsm · 2020-07-28T14:21:16Z

I filed radar bug FB8190988 on Apple's system with the ranlib/ar warning logs and repro instructions from this PR.

xavierleroy

I reviewed again. Looks good to me. I see no reason to delay merging, since the remaining macOS problems ("ar" warnings and "tests/unwind" failure) are probably not our fault and can be worked around later.

shindere · 2020-07-29T10:00:09Z

Am I correct that the INSTALL file has not been updated and that we don't
currently have any documentation explaining how to build OCaml on iOS
in the repository?

EduardoRFS · 2020-07-29T10:17:28Z

@shindere you would be correct, but is the same as any compiler after the fix on config.guess, if you're using jailbreak. And is the the same as any cross compiler otherwise. Just configure and make

shindere · 2020-07-29T10:26:00Z

OK ok, and I guess since deploying includes jailbreak we can not realy document the procedure, that's too bad.

m-schmidt · 2020-07-29T21:04:52Z

I filed radar bug FB8190988 ...

Mine is FB8207693 (which makes 16k filed bugs in under 2 days). I made a shorter example to reproduce with just plain C files. Let's hope for the best.

m-schmidt · 2020-08-06T22:40:26Z

Good news! With Beta 4 the alignment warnings of ar/ranlib are gone.

Add support for iOS and macOS on ARM64

EduardoRFS force-pushed the trunk-ios branch from a53e47b to 283f3a6 Compare June 22, 2020 01:24

XVilka mentioned this pull request Jun 22, 2020

iOS support for OCaml #6986

Closed

avsm reviewed Jun 22, 2020

View reviewed changes

runtime/sys.c Outdated Show resolved Hide resolved

EduardoRFS force-pushed the trunk-ios branch from 972bd70 to a48e185 Compare June 24, 2020 18:47

xavierleroy reviewed Jun 29, 2020

View reviewed changes

runtime/arm64.S Show resolved Hide resolved

xavierleroy reviewed Jun 29, 2020

View reviewed changes

EduardoRFS changed the title ~~Add support to iOS / Darwin ARM64~~ Add support to iOS / Mac ARM64 Jul 2, 2020

xavierleroy approved these changes Jul 28, 2020

View reviewed changes

xavierleroy merged commit a0a1ba4 into ocaml:trunk Jul 28, 2020

xavierleroy added a commit that referenced this pull request Jul 28, 2020

Changes entry for #9699

274cb3c

EduardoRFS deleted the trunk-ios branch July 28, 2020 15:05

claui mentioned this pull request Jul 28, 2020

macOS 11 Big Sur compatibility on Apple Silicon Homebrew/brew#7857

Closed

9 tasks

bryphe mentioned this pull request Sep 3, 2020

ARM64 support onivim/oni2#2400

Open

EduardoRFS pushed a commit to EduardoRFS/ocaml that referenced this pull request Sep 8, 2020

Merge pull request ocaml#9699 from EduardoRFS/trunk-ios

c9992b3

Add support for iOS and macOS on ARM64

EduardoRFS pushed a commit to EduardoRFS/ocaml that referenced this pull request Sep 8, 2020

Merge pull request ocaml#9699 from EduardoRFS/trunk-ios

944d9ad

Add support for iOS and macOS on ARM64

enieber mentioned this pull request Sep 14, 2020

Edition 0 enieber/re-newsletter#1

Closed

EduardoRFS pushed a commit to EduardoRFS/ocaml that referenced this pull request Nov 16, 2020

Merge pull request ocaml#9699 from EduardoRFS/trunk-ios

497bdda

Add support for iOS and macOS on ARM64

EduardoRFS pushed a commit to EduardoRFS/ocaml that referenced this pull request Nov 16, 2020

Merge pull request ocaml#9699 from EduardoRFS/trunk-ios

eecf8fe

Add support for iOS and macOS on ARM64

EduardoRFS mentioned this pull request Nov 16, 2020

Backport macOS ARM to 4.10 #10026

Merged

EduardoRFS pushed a commit to EduardoRFS/ocaml that referenced this pull request Nov 18, 2020

Merge pull request ocaml#9699 from EduardoRFS/trunk-ios

ba11dab

Add support for iOS and macOS on ARM64

avsm pushed a commit to avsm/ocaml that referenced this pull request Nov 26, 2020

Merge pull request ocaml#9699 from EduardoRFS/trunk-ios

6807bb0

Add support for iOS and macOS on ARM64

EduardoRFS pushed a commit to EduardoRFS/ocaml that referenced this pull request Nov 30, 2020

Merge pull request ocaml#9699 from EduardoRFS/trunk-ios

d6a77c1

Add support for iOS and macOS on ARM64

EduardoRFS pushed a commit to EduardoRFS/ocaml that referenced this pull request Dec 2, 2020

Merge pull request ocaml#9699 from EduardoRFS/trunk-ios

7b9612f

Add support for iOS and macOS on ARM64

ygrek mentioned this pull request Dec 9, 2020

Changes: fix typo in PR number #10079

Closed

enieber mentioned this pull request Jan 28, 2021

Edition 18/02/2021 enieber/re-newsletter#8

Closed

bobrik mentioned this pull request Feb 27, 2021

ocamlPackages.ocaml: 4.10.0 -> 4.10.2 NixOS/nixpkgs#114526

Merged

10 tasks

enieber mentioned this pull request Mar 1, 2021

Edition 18/03/2021 enieber/re-newsletter#9

Closed

gasche mentioned this pull request Nov 29, 2024

Fix a few typos #13652

Merged

Add support to iOS / Mac ARM64 #9699

Add support to iOS / Mac ARM64 #9699

Conversation

EduardoRFS commented Jun 22, 2020 • edited Loading

Requirements

About this PR

Changes to Linux ARM64

Current state + Tests

EduardoRFS commented Jun 22, 2020

shindere commented Jun 22, 2020 via email

EduardoRFS commented Jun 23, 2020

xavierleroy commented Jun 24, 2020

EduardoRFS commented Jun 24, 2020

m-schmidt commented Jun 29, 2020 • edited Loading

xavierleroy commented Jun 29, 2020

xavierleroy left a comment

Choose a reason for hiding this comment

xavierleroy Jun 29, 2020

Choose a reason for hiding this comment

EduardoRFS Jun 29, 2020

Choose a reason for hiding this comment

xavierleroy left a comment

Choose a reason for hiding this comment

xavierleroy left a comment

Choose a reason for hiding this comment

xavierleroy left a comment

Choose a reason for hiding this comment

xavierleroy left a comment • edited Loading

Choose a reason for hiding this comment

xavierleroy left a comment

Choose a reason for hiding this comment

xavierleroy left a comment

Choose a reason for hiding this comment

xavierleroy Jun 29, 2020

Choose a reason for hiding this comment

EduardoRFS Jun 29, 2020 • edited Loading

Choose a reason for hiding this comment

EduardoRFS commented Jun 29, 2020 • edited Loading

EduardoRFS commented Jun 29, 2020 • edited Loading

m-schmidt commented Jun 30, 2020

EduardoRFS commented Jun 30, 2020

xavierleroy commented Jun 30, 2020

shindere commented Jun 30, 2020 via email

shindere commented Jun 30, 2020 via email

EduardoRFS commented Jun 30, 2020

shindere commented Jun 30, 2020 via email

EduardoRFS commented Jun 30, 2020 • edited Loading

xavierleroy commented Jul 1, 2020 • edited Loading

avsm commented Jul 27, 2020 • edited Loading

xavierleroy commented Jul 28, 2020

xavierleroy commented Jul 28, 2020

avsm commented Jul 28, 2020

xavierleroy left a comment

Choose a reason for hiding this comment

shindere commented Jul 29, 2020

EduardoRFS commented Jul 29, 2020

shindere commented Jul 29, 2020 via email

m-schmidt commented Jul 29, 2020

m-schmidt commented Aug 6, 2020

EduardoRFS commented Jun 22, 2020 •

edited

Loading

m-schmidt commented Jun 29, 2020 •

edited

Loading

xavierleroy left a comment •

edited

Loading

EduardoRFS Jun 29, 2020 •

edited

Loading

EduardoRFS commented Jun 29, 2020 •

edited

Loading

EduardoRFS commented Jun 29, 2020 •

edited

Loading

EduardoRFS commented Jun 30, 2020 •

edited

Loading

xavierleroy commented Jul 1, 2020 •

edited

Loading

avsm commented Jul 27, 2020 •

edited

Loading