Support remote compilation #622

waruqi · 2019-11-19T03:59:59Z

Roadmap

socket io and coroutine scheduler
xmake connect xxx.xxx.xxx.xxx
xmake (remote build)
xmake config (remote configuration)
xmake run (remote run)

Refs

#165 (comment)
#593

arkena00 · 2022-02-15T17:21:52Z

+1 for this

I was thinking about something like this :

xmake build --host host.com --user me --pass security

or using a custom toolchain maybe

xmake build --toolchain=gcc@host

toolchain("gcc@host")
    set_kind("remote")
    set_user("me")
    ...

I'm not sure if it's possible to make it simple to integrate in an IDE if you have to run xmake connect first

waruqi · 2022-02-15T23:38:43Z

This feature need take a lot of time and work, and maybe I'll start supporting it again in the future, but not now.

Of course, if more people request this feature, I'll also consider it when I have time.

SongZihui-sudo · 2022-04-08T03:48:38Z

做一个插件,可不可以用git来同步远程与本地的源文件，之后在远程建一个socket服务器来接受编译的命令，编译完成，在本地用pull同步结果。

Make a plugin,Maybe we can use 'git' to synchronize remote and local source files, and then run a 'socket server' remotely to accept compiled commands, etc., compile finish ，then we can synchronize the results with 'pull' locally.

waruqi · 2022-04-08T04:13:16Z

做一个插件,可不可以用git来同步远程与本地的源文件，之后在远程建一个socket服务器来接受编译的命令，编译完成，在本地用pull同步结果。

Make a plugin,Maybe we can use 'git' to synchronize remote and local source files, and then run a 'socket server' remotely to accept compiled commands, etc., compile finish ，then we can synchronize the results with 'pull' locally.

现在的问题不是怎么做，是我没时间精力做

waruqi · 2022-04-15T15:49:49Z

Start service

$ xmake service
<remote_build_server>: listening 0.0.0.0:90091 ..

with verbose logs

$ xmake service -vD
<remote_build_server>: listening 0.0.0.0:90091 ..

Start and stop service with daemon mode

$ xmake service --start
$ xmake service --restart
$ xmake service --stop

Configure service on server side

run xmake service will generate a default service.conf file in ~/.xmake/service.conf

edit it and modify server user name for using git push to sync files.

{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        server = {
            listen = "0.0.0.0:90091"
        }
    }
}

Configure service on client side

~/.xmake/service.conf

{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        client = {
            connect = "192.168.56.101:90091",
        }
    }
}

Import the given configuration file

$ xmake service --config=/tmp/service.conf

Connect remote build service on client side

enter project directory

$ xmake create test
$ cd test
$ xmake service --connect 
<remote_build_client>: connect 192.168.56.110:90091 ..
<remote_build_client>: connected!
<remote_build_client>: sync files in 192.168.56.110:90091 ..
Scanning files ..
Comparing 3 files ..
    [+]: src/main.cpp
    [+]: .gitignore
    [+]: xmake.lua
3 files has been changed!
Archiving files ..
Uploading files with 1372 bytes ..
<remote_build_client>: sync files ok!

Build project with remote mode

$ xmake
<remote_build_client>: run xmake in 192.168.56.110:90091 ..
checking for platform ... macosx
checking for architecture ... x86_64
checking for Xcode directory ... /Applications/Xcode.app
checking for Codesign Identity of Xcode ... Apple Development: waruqi@gmail.com (T3NA4MRVPU)
checking for SDK version of Xcode for macosx (x86_64) ... 11.3
checking for Minimal target version of Xcode for macosx (x86_64) ... 11.4
[ 25%]: ccache compiling.release src/main.cpp
[ 50%]: linking.release test
[100%]: build ok!
<remote_build_client>: run command ok!

Run target with remote mode

$ xmake run
<remote_build_client>: run xmake run in 192.168.56.110:90091 ..
hello world!
<remote_build_client>: run command ok!

Rebuild target with remote mode

$ xmake -rv
<remote_build_client>: run xmake -rv in 192.168.56.110:90091 ..
[ 25%]: ccache compiling.release src/main.cpp
/usr/local/bin/ccache /usr/bin/xcrun -sdk macosx clang -c -Qunused-arguments -arch x86_64 -mmacosx-version-min=11.4 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -o build/.objs/test/macosx/x86_64/release/src/main.cpp.o src/main.cpp
[ 50%]: linking.release test
"/usr/bin/xcrun -sdk macosx clang++" -o build/macosx/x86_64/release/test build/.objs/test/macosx/x86_64/release/src/main.cpp.o -arch x86_64 -mmacosx-version-min=11.4 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk -stdlib=libc++ -Wl,-x -lz
[100%]: build ok!
<remote_build_client>: run command ok!

Configure project with remote mode

$ xmake f --xxx --yy

Sync files to remote server

use git to commit code and run sync

$ git add .
$ git commit -a -m "..."
$ xmake service --sync 
<remote_build_client>: sync files in 192.168.56.110:90091 ..
Scanning files ..
Comparing 3 files ..
    [+]: src/main.cpp
    [+]: .gitignore
    [+]: xmake.lua
3 files has been changed!
Archiving files ..
Uploading files with 1372 bytes ..
<remote_build_client>: sync files ok!

Disconnect remote service for the current project

$ xmake service --disconnect
<remote_build_client>: disconnect 192.168.56.110:90091 ..
<remote_build_client>: disconnected!

View service logs

$ xmake service --logs

Clean all built and cache files for the current project on server

$ cd projectdir
$ xmake service --clean

TODO

Pull built target files to local host

waruqi · 2022-04-15T15:58:04Z

I have removed git/ssh dependences

Some issues related to git/OpenSSH server on windows

https://github.com/PowerShell/Win32-OpenSSH/wiki/Setting-up-a-Git-server-on-Windows-using-Git-for-Windows-and-Win32_OpenSSH

Set powershell as default shell

Fix git push ruki@host:C:/xx/test failed, it will cause that sync files failure.
New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force
New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShellCommandOption -Value "/c" -PropertyType String -Force
Set git/mingw64/bin to %PATH%

to solve git-receive-pack not found!
$gitPath = Join-Path -Path $env:ProgramFiles -ChildPath "git\mingw64\bin"
$machinePath = [Environment]::GetEnvironmentVariable('Path', 'MACHINE')
[Environment]::SetEnvironmentVariable('Path', "$gitPath;$machinePath", 'Machine')
FIXME

it still be failure when we run git push to windows server. : (
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 1.25 KiB | 638.00 KiB/s, done.
Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly

waruqi · 2022-04-16T13:46:01Z

I have supported it.

xq114 · 2022-04-18T12:28:38Z

Can I build the project on the remote machine, but run it on my local machine? How can I install the built binaries on localhost?

waruqi · 2022-04-18T12:39:34Z

Can I build the project on the remote machine, but run it on my local machine? How can I install the built binaries on localhost?

Build project and run target on the remote machine. And we can also pull built target to the local machine (but I have not supported it yet.)

waruqi · 2022-04-30T10:02:22Z

User authorization

At the moment I don't have time to do perfect encryption and authentication based on ssl, but I have implemented some basic user authorization authentication. On a LAN it should also avoid most of the security problems.

We can support two authentication modes

User password mode
Token mode

Token mode

Generate default configuration in server

We run xmake service to generate a default configuraion. It will generate an initial token.

$ xmake service
generating the config file to /Users/ruki/.xmake/service.conf ..
an token(4b928c7563a0cba10ff4c3f5ca0c8e24) is generated, we can use this token to connect servi
ce.
$ cat /Users/ruki/.xmake/service.conf
{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        client = {
            connect = "root@127.0.0.1:9691",
            token = "4b928c7563a0cba10ff4c3f5ca0c8e24"
        },
        server = {
            listen = "0.0.0.0:9691",
            workdir = "/Users/ruki/.xmake/service/remote_build"
        }
    },
    server = {
        tokens = {
            "4b928c7563a0cba10ff4c3f5ca0c8e24"
        },
        known_hosts = { }
    }
}

Add token in client

Then we add this token in the client configuraion.

$ cat /Users/ruki/.xmake/service.conf
{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        client = {
            connect = "root@127.0.0.1:9691",
            token = "4b928c7563a0cba10ff4c3f5ca0c8e24"
        }
    }
}

Connect to server

Now we can connect to server to build project.

$ xmake service --connect
<remote_build_client>: connect 127.0.0.1:9691 ..
<remote_build_client>: connected!
<remote_build_client>: sync files in 127.0.0.1:9691 ..
Scanning files ..
Comparing 3 files ..
    [+]: xmake.lua
    [+]: .gitignore
    [+]: src/main.cpp
3 files has been changed!
Archiving files ..
Uploading files with 1591 bytes ..
<remote_build_client>: sync files ok!

Generate a new token

We can also generate a new token for new client.

$ xmake service --gen-token
New token a7b9fc2d3bfca1472aabc38bb5f5d612 is generated!

Then we can copy and save this token to new client configuraion file.

User password mode

Add user in server

$ xmake service --add-user=ruki
Please input user ruki password:
123456
Add user ruki ok!

$ cat /Users/ruki/.xmake/service.conf
{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        client = {
            connect = "root@127.0.0.1:9691"
        },
        server = {
            listen = "0.0.0.0:9691",
            workdir = "/Users/ruki/.xmake/service/remote_build"
        }
    },
    server = {
        tokens = {
            "4b928c7563a0cba10ff4c3f5ca0c8e24",
            "7889e25402413e93fd37395a636bf942"
        },
        known_hosts = { }
    }
}

Remove user in server

$xmake service --rm-user=ruki
Please input user ruki password:
123456
Remove user ruki ok!

Configurate user in client

We can add user name to the connect address. eg. user@address:port

{
    remote_build = {
        client = {
            connect = "root@127.0.0.1:9691"
        }
  }
}

Connect to server

We need input user password before connecting server.

$ xmake service --connect
Please input user root password:
000000
<remote_build_client>: connect 127.0.0.1:9691 ..
<remote_build_client>: connected!
<remote_build_client>: sync files in 127.0.0.1:9691 ..
Scanning files ..
Comparing 3 files ..
    [+]: xmake.lua
    [+]: .gitignore
    [+]: src/main.cpp
3 files has been changed!
Archiving files ..
Uploading files with 1591 bytes ..
<remote_build_client>: sync files ok!

If password is incorrect, it will raise errors.

 xmake service --connect
Please input user root password:
123
<remote_build_client>: connect 127.0.0.1:9691 ..
<remote_build_client>: connect 127.0.0.1:9691 failed, user and password are incorrect!

Configure known hosts in server

We can also add known hosts in server to guarantee the connection of trusted hosts.

We can add them to the known_hosts = {} in service.conf

$ cat ~/.xmake/service.conf 
{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    server = {
        tokens = {
            "4b928c7563a0cba10ff4c3f5ca0c8e24"
        },
        known_hosts = { "127.0.0.1", "xx.xx.xx.xx"}
    }
}

If it is configured, then only hosts included in this list will be able to successfully connect to the server.

waruqi · 2022-05-15T08:24:38Z

To better manage the configuration, I split service.conf into two configuration files, server.conf and client.conf

$ cat ~/.xmake/service/server.conf 
{
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    remote_build = {
        listen = "0.0.0.0:9691",
        workdir = "/Users/ruki/.xmake/service/server/remote_build"
    },
    tokens = {
        "e438d816c95958667747c318f1532c0f"
    }
}

$ cat ~/.xmake/service/client.conf 
{
    remote_build = {
        connect = "127.0.0.1:9691",
        token = "e438d816c95958667747c318f1532c0f"
    }
}

waruqi added the feature request label Nov 19, 2019

waruqi added this to the todo milestone Nov 19, 2019

waruqi pinned this issue Nov 20, 2019

waruqi modified the milestones: todo, v2.3.0 Dec 20, 2019

waruqi modified the milestones: v2.3.0, v2.3.1 Feb 21, 2020

waruqi modified the milestones: v2.3.2, v2.3.3 Mar 24, 2020

waruqi modified the milestones: v2.3.3, v2.3.4 Apr 17, 2020

waruqi modified the milestones: v2.3.4, todo May 13, 2020

waruqi unpinned this issue May 15, 2020

waruqi modified the milestones: todo, v2.6.5 Apr 15, 2022

waruqi mentioned this issue Apr 15, 2022

Generate linux project for visual studio 2017. #165

Closed

waruqi closed this as completed Apr 16, 2022

waruqi mentioned this issue May 15, 2022

Distributed compilation support #274

Closed

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support remote compilation #622

Support remote compilation #622

waruqi commented Nov 19, 2019 •

edited

Loading

arkena00 commented Feb 15, 2022 •

edited

Loading

waruqi commented Feb 15, 2022 •

edited

Loading

SongZihui-sudo commented Apr 8, 2022

waruqi commented Apr 8, 2022

waruqi commented Apr 15, 2022 •

edited

Loading

waruqi commented Apr 15, 2022 •

edited

Loading

Set powershell as default shell

Set git/mingw64/bin to %PATH%

FIXME

waruqi commented Apr 16, 2022

xq114 commented Apr 18, 2022

waruqi commented Apr 18, 2022

waruqi commented Apr 30, 2022 •

edited

Loading

waruqi commented May 15, 2022

Support remote compilation #622

Support remote compilation #622

Comments

waruqi commented Nov 19, 2019 • edited Loading

Roadmap

Refs

arkena00 commented Feb 15, 2022 • edited Loading

waruqi commented Feb 15, 2022 • edited Loading

SongZihui-sudo commented Apr 8, 2022

waruqi commented Apr 8, 2022

waruqi commented Apr 15, 2022 • edited Loading

Start service

Start and stop service with daemon mode

Configure service on server side

Configure service on client side

Import the given configuration file

Connect remote build service on client side

Build project with remote mode

Run target with remote mode

Rebuild target with remote mode

Configure project with remote mode

Sync files to remote server

Disconnect remote service for the current project

View service logs

Clean all built and cache files for the current project on server

TODO

waruqi commented Apr 15, 2022 • edited Loading

Set powershell as default shell

Set git/mingw64/bin to %PATH%

FIXME

waruqi commented Apr 16, 2022

xq114 commented Apr 18, 2022

waruqi commented Apr 18, 2022

waruqi commented Apr 30, 2022 • edited Loading

User authorization

Token mode

Generate default configuration in server

Add token in client

Connect to server

Generate a new token

User password mode

Add user in server

Remove user in server

Configurate user in client

Connect to server

Configure known hosts in server

waruqi commented May 15, 2022

waruqi commented Nov 19, 2019 •

edited

Loading

arkena00 commented Feb 15, 2022 •

edited

Loading

waruqi commented Feb 15, 2022 •

edited

Loading

waruqi commented Apr 15, 2022 •

edited

Loading

waruqi commented Apr 15, 2022 •

edited

Loading

waruqi commented Apr 30, 2022 •

edited

Loading