Windows version #1540

mxmauro · 2020-09-22T18:49:39Z

This commit makes modifications to source code to be able to generate native Windows binaries with the help of MSYS2 build environment.

Some of the original shell scripts for building the apps were sightly modified also to work under Windows environments.

The instructions, also located in /scripts/windows/instructions.md to build the executable files are:

Download and install MSYS2 package from here
Run MSYS2 MingW 64-bit application to open the MSYS2 terminal.
Update MSYS2 package and dependency manager by running the following commands:
```
pacman -Syu --disable-download-timeout
```
NOTE: It is very likely MSYS2 will ask to close the window and repeat the command for furter updates. Check MSYS2 web page for additional support.
Install GIT on MSYS2 by executing the following command:
```
pacman -S --disable-download-timeout --noconfirm git
```
Clone repository with git clone https://github.com/algorand/go-algorand
Switch to source code directory with cd go-algorand.
Run ./scripts/windows/install_deps.sh to install required dependencies.
Run make.

In /home/{you-user}/go/bin you will find all the apps. There is no installer package right know but you can copy genesis.json and create a config.json under data subdirectory and run goal node start -d data as you would do under other OSes.

Fork update

Update master to Sep-21

tsachiherman

If we'll put aside that build environment changes, most of the go-code changes looks very well written and I don't have any reason to block these.
Unfortunately, I don't have the setup to test the build environment, so I can't approve this change. Maybe with some additional changes to the travis automated build, I could see how this compile successfully.

crypto/memcpy_chk_windows.c

libgoal/lockedFileWindows.go

util/util_windows.go

mxmauro · 2020-09-22T21:09:07Z

Hi @tsachiherman thanks for the feedback. I'll check it better and upload fixes. BTW: Do you know the reason of the golint error Travis is throwing?

mxmauro · 2020-09-23T11:53:56Z

Hi @tsachiherman finally I could get rid of the issues.

Something weird I found, util_windows.go has a wrong license header but check_license.sh script started to modify all .go files because of this.

I don't how to setup Travis CI. But I asked for assistance to @rfustino to also check on Windows. Should be straight-forward. Just Windows 10 + MSYS2 and running a script to install dependencies.

Of course I can assist in whatever you need.

util/util_windows.go

cmd/kmd/mlock_windows.go

logging/usage_windows.go

…ndows.

algonautshant

Looks very nice.
I have some comments/questions.

algonautshant · 2020-09-25T15:47:42Z

Makefile

+UNAME := $(shell uname)
+ifneq (, $(findstring MINGW,$(UNAME)))
+	#Gopath is not saved across sessions, probably existing Windows env vars, override them
+	export GOPATH := ${HOME}/go


On Windows (depending on the go compiler used), Go expects GOPATH to be an absolute path with the format <drive>:<path>
What will ${HOME} evaluate to here?

Hi @algonautshant , The whole build process, including the installation of Go, runs inside MSYS2 ecosystem. HOME defaults to /home/{your-windows-username}.

Although you can access the whole disk with /c/..., /home and the rest of the common Unix/Linux directories are located inside MSYS2 installation path.

algonautshant · 2020-09-25T15:49:16Z

Makefile

-export GOPATH		:= $(shell go env GOPATH)
-GOPATH1		:= $(firstword $(subst :, ,$(GOPATH)))
+UNAME := $(shell uname)
+ifneq (, $(findstring MINGW,$(UNAME)))


Instead of evaluating ifneq (, $(findstring MINGW,$(UNAME))) every time, would be clearer to make a boolean constant for it, and check it here and elsewhere.

Yes, may be. Bash and Linux is not my strong suite. :)

algonautshant · 2020-09-25T15:49:38Z

Makefile

@@ -38,6 +45,10 @@ endif
 endif
 endif

+ifneq (, $(findstring MINGW,$(UNAME)))
+EXTLDFLAGS := -static-libstdc++ -static-libgcc


Indentation.

Do you prefer to add to this line or remove the ones at the beginning like the rest of the file?

I like the indentation. I prefer adding indentation here as well. Thanks.

SInce this is ifneq, this will overwrite the EXTLDFLAGS set for linux in the previous if block won't it?

this is a tricky expression -
on linux, the following expression $(findstring MINGW,$(UNAME)) would evaluate to an empty string, right ?
if so, the expression ifeq (, <empty string>) is clearly true, and the ifneq (, <empty string>) is false.

On the other hand, on windows, it's the opposite. A comment above could clarify things up.

It is a similar approach to the one used to test flags: https://www.gnu.org/software/make/manual/make.html#Testing-Flags

network/wsNetwork_windows.go

algonautshant · 2020-09-27T06:14:04Z

scripts/configure_dev.sh

+        exit 1
+    fi
+
+    export GOPATH=$HOME/go


Why configure_dev needs to export GOPATH here (only for windows)?

I completely forgot about an old attempt to move all the actions of the /scripts/windows directory inside configure_dev.sh to normalize set up of development environment. I'll fix it.

About exporting GOPATH, because the whole process runs under some sort of virtual environment, I assume Go is not installed and I need it to run configure_dev-deps.sh, install shellcheck (there is no package available under MSYS2, and copy libwinpthread-1.dll (cannot compile statically because of karalabe/hid dependency using libusb)

algonautshant · 2020-09-27T06:23:25Z

scripts/configure_dev.sh

@@ -41,6 +41,45 @@ function install_or_upgrade {
    fi
 }

+function install_windoows_shellcheck() {


This function is not needed here, since a file is added for the same purpose: scripts/windows/install_shellcheck.sh

CLAassistant · 2020-09-28T16:41:29Z

All committers have signed the CLA.

mxmauro · 2020-09-28T18:43:08Z

Hi @algonautshant , I moved the installation of dependencies to the configure_dev.sh script and also merged almost the latest changes from master.

Despite of this, I'm a bit lost on the failed Travis build about external arm64.

algonautshant · 2020-09-29T02:21:57Z

@mxmauro your job is good. It was one of the tests timed-out. A rerun passed.

mxmauro · 2020-09-29T02:45:10Z

Thanks @algonautshant. Next step would be to create Travis support and an installation package using Wix (although need to see how to integrate Wix Toolset and MSYS2)

scripts/configure_dev.sh

tsachiherman · 2020-09-29T13:43:27Z

scripts/configure_dev.sh

+    fi
+
+    # This is required because http://github.com/karalabe/hid library compiles with non-static libraries
+    cp /mingw64/bin/libwinpthread-1.dll $GOPATH/bin/ 


I think it would be cleaner to add the /mingw64/bin directory to the path on compiling machines.
For packaged deployment, we need to copy the above file anyway.

Actually is already there when you open the MSYS2 MingW64 terminal:

This is my current configuration:

Blaster64@BlasterNotebook MINGW64 ~ $ echo $PATH /mingw64/bin:/usr/local/bin:/usr/bin:/bin:/c/Windows/System32:/c/Windows:/c/Windows/System32/Wbem:/c/Windows/System32/WindowsPowerShell/v1.0/:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl

if it's already there, why do you need to copy it ? windows DLL loader would already search on the PATH, and would find it. Is there any usage of this DLL as a library source file ? i.e. building the stub out of an existing DLL ?

Because you usually don't run the app inside MSYS2. The goal is to create executable files that can be distributed and used anywhere.

Despite it is a simple wrapper to convert pthread api into native Windows ones, I'll continue trying to remove that dependency and see if I can compile the whole code statically.

About source code, here: https://github.com/mirror/mingw-w64/tree/master/mingw-w64-libraries/winpthreads

nodecontrol/NodeController.go

tsachiherman · 2020-09-29T14:09:38Z

util/process_windows.go

+)
+
+// KillProcess kills a running OS process
+func KillProcess(pid int, _ os.Signal) error {


This implementation is a great fit for the os.Signal SIGKILL. It doesn't emulate the SIGTERM correctly on windows.
Implementing it correctly require some extra work ( which can be deferred to a follow-up PR ):

Add a hidden console window to the algod and configure a SetConsoleCtrlHandler

In KillProcess, provide handling for SIGINT:
Find the window that is associated with the process, and use GenerateConsoleCtrlEvent

In KillProcess, provide handling for SIGTERM:
Find the window that is associated with the process, and send WM_QUIT using SendMessage.

Go already sets up a SetConsoleCtrlHandler but the problem is how to send the signal to the target process.

Workarounds includes injecting a remote thread or attaching the target console (I don't like this approach)

Whatever the method, it will have some complexity.

tsachiherman

looks great, thank you for your contribution.

mxmauro · 2020-09-29T23:46:00Z

You are welcome.

Next steps:

Investigate about Travis CI
If suits for you, use github.com/kardianos/service or build some sort of helper app to be able to launch algod as a service.
Installer package.

Add support for algod to compile natively under Windows.

mxmauro and others added 5 commits February 15, 2020 17:12

Merge pull request #1 from algorand/master

e884e13

Fork update

Merge pull request #2 from algorand/master

ad1cf07

Update master to Sep-21

Source code modifications to make them compilable under Windows

0cc6f1c

Scripts to setup development environment under MSYS2

12d0f07

Moved dll dependency copy at the end of setup script

f24f931

tsachiherman reviewed Sep 22, 2020

View reviewed changes

crypto/memcpy_chk_windows.c Outdated Show resolved Hide resolved

libgoal/lockedFileWindows.go Outdated Show resolved Hide resolved

util/util_windows.go Outdated Show resolved Hide resolved

tsachiherman requested review from algonautshant, algorandskiy, brianolson, winder and onetechnical September 22, 2020 19:32

Fixed Linux compilation issues.

627e07d

mxmauro added 9 commits September 22, 2020 19:28

Removed raise exception after cgo_painc call

5eeb7f5

Removed dummy comments

fb401cb

Added comments to exported functions

df38db3

Fixed build exception on intended target OS

8235d7c

Added common functions for all OSes

e40994f

Fixed code formatting

71aa8ad

Added missing return at end of function

981a7d5

Removed underscore from variables

f6cd990

Changed shell engine to newer bash

5f6538e

tsachiherman reviewed Sep 23, 2020

View reviewed changes

util/util_windows.go Show resolved Hide resolved

tsachiherman reviewed Sep 23, 2020

View reviewed changes

cmd/kmd/mlock_windows.go Outdated Show resolved Hide resolved

tsachiherman reviewed Sep 23, 2020

View reviewed changes

logging/usage_windows.go Outdated Show resolved Hide resolved

mxmauro added 2 commits September 23, 2020 13:47

Added a common function to get process running times.

ae02e07

Added log and comment because of lack of mlockall functionality on Wi…

5dbcda3

…ndows.

algonautshant reviewed Sep 27, 2020

View reviewed changes

Removed indentation

08fb191

mxmauro and others added 4 commits September 28, 2020 10:08

Merged Windows dependencies installer script with configure_dev.sh

eb48485

Updated Windows environment setup instructions

d4de8ab

Merge remote-tracking branch 'upstream/master'

1d45ba4

Merge branch 'master' into windows-version

8d3db9a

algonautshant previously approved these changes Sep 28, 2020

View reviewed changes

tsachiherman reviewed Sep 29, 2020

View reviewed changes

scripts/configure_dev.sh Outdated Show resolved Hide resolved

tsachiherman reviewed Sep 29, 2020

View reviewed changes

nodecontrol/NodeController.go Outdated Show resolved Hide resolved

Fixed wrong change of SIGKILL signal to SIGTERM

41087ef

mxmauro dismissed algonautshant’s stale review via 41087ef September 29, 2020 13:54

Fixed typo configure dev script

c1a10ec

tsachiherman reviewed Sep 29, 2020

View reviewed changes

tsachiherman approved these changes Sep 29, 2020

View reviewed changes

tsachiherman merged commit 0bb4cf3 into algorand:master Sep 29, 2020

mxmauro deleted the windows-version branch September 30, 2020 01:03

tsachiherman pushed a commit to tsachiherman/go-algorand that referenced this pull request Jul 7, 2021

Windows version (algorand#1540)

b9c9e34

Add support for algod to compile natively under Windows.

PhearZero pushed a commit to PhearNet/crypto that referenced this pull request Jan 17, 2025

Windows version (algorand#1540)

1375acc

Add support for algod to compile natively under Windows.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Windows version #1540

Windows version #1540

mxmauro commented Sep 22, 2020 •

edited

Loading

tsachiherman left a comment

mxmauro commented Sep 22, 2020

mxmauro commented Sep 23, 2020

algonautshant left a comment

algonautshant Sep 25, 2020

mxmauro Sep 27, 2020

algonautshant Sep 28, 2020

algonautshant Sep 25, 2020

mxmauro Sep 27, 2020

algonautshant Sep 25, 2020

mxmauro Sep 27, 2020

algonautshant Sep 28, 2020

bricerisingalgorand Sep 29, 2020

tsachiherman Sep 29, 2020 •

edited

Loading

mxmauro Sep 29, 2020 •

edited

Loading

algonautshant Sep 27, 2020

mxmauro Sep 27, 2020

algonautshant Sep 27, 2020

mxmauro Sep 29, 2020

CLAassistant commented Sep 28, 2020 •

edited

Loading

mxmauro commented Sep 28, 2020

algonautshant commented Sep 29, 2020

mxmauro commented Sep 29, 2020

tsachiherman Sep 29, 2020

mxmauro Sep 29, 2020

tsachiherman Sep 29, 2020

mxmauro Sep 29, 2020

tsachiherman Sep 29, 2020

mxmauro Sep 29, 2020

tsachiherman left a comment

mxmauro commented Sep 29, 2020

Windows version #1540

Windows version #1540

Conversation

mxmauro commented Sep 22, 2020 • edited Loading

tsachiherman left a comment

Choose a reason for hiding this comment

mxmauro commented Sep 22, 2020

mxmauro commented Sep 23, 2020

algonautshant left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsachiherman Sep 29, 2020 • edited Loading

Choose a reason for hiding this comment

mxmauro Sep 29, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

CLAassistant commented Sep 28, 2020 • edited Loading

mxmauro commented Sep 28, 2020

algonautshant commented Sep 29, 2020

mxmauro commented Sep 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tsachiherman left a comment

Choose a reason for hiding this comment

mxmauro commented Sep 29, 2020

mxmauro commented Sep 22, 2020 •

edited

Loading

tsachiherman Sep 29, 2020 •

edited

Loading

mxmauro Sep 29, 2020 •

edited

Loading

CLAassistant commented Sep 28, 2020 •

edited

Loading