Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can cdc_rsync be used as a local Windows rsync tool? #61

Open
prarsena opened this issue Jan 11, 2023 · 10 comments
Open

Can cdc_rsync be used as a local Windows rsync tool? #61

prarsena opened this issue Jan 11, 2023 · 10 comments
Labels
question Further information is requested

Comments

@prarsena
Copy link

I saw the following in another issue thread :

adding support for Windows to Windows cdc_rsync (including local syncs, i.e. not over network), see
main...sync_windows

Between the speed improvements you highlight from this tool and the fact that Windows doesn't have native support for rsync, I would love to use this tool to make backups for local drives.

But I wasn't able to understand the status of that work. Is the current status of the sync_windows (or win_support) branch at the point to support local Win rsync?

If the support comes, would it work like similarly to the following format? cdc_rsync E:\ F:\MyDrive\ -rv

@ljusten
Copy link
Collaborator

ljusten commented Jan 11, 2023

The status is, the code is in review. I expect it to land next week. My reviewer is currently on vacation.

If the support comes, would it work like similarly to the following format? cdc_rsync E:\ F:\MyDrive\ -rv
Yes, exactly.

I somewhat doubt, though, that cdc_rsync will perform any better than standard Windows tools like xcopy and robocopy in this case. Note that both tools can skip unchanged files by modification date. cdc_rsync only gives a benefit if the bandwidth between the local and the target system is lower than the disk IO speeds on the two systems. In case of a local copy the bandwidth is very high (basically, memory speed), so I don't expect any gain.

To illustrate what I mean, imagine there's a 1 GB file F:\MyDrive\foo.dat and the same file with a single byte inserted in the beginning at E:\foo.dat. In order to sync this file, cdc_rsync creates a temp file F:\MyDrive\foo.dat__cdc_rsync_temp__, copies a chunk of E:\foo.dat to it and appends the rest from F:\MyDrive\foo.dat. Finally, it replaces F:\MyDrive\foo.dat by F:\MyDrive\foo.dat__cdc_rsync_temp__. I highly doubt this is any faster than a plain copy.

I would suggest you try it out once the changes are in, and compare. I'm very curious what the results are.

@ljusten ljusten added the question Further information is requested label Jan 11, 2023
@prarsena
Copy link
Author

Thank you for your detailed response, and explaining that the benefit of cdc_rsync comes from the network improvements. I appreciate it. I will keep an eye on this project and test that out.

@rstupek
Copy link

rstupek commented Jan 17, 2023

When would a test version for windows->windows remote machine be available for testing? I would be willing to give it a try

@ljusten
Copy link
Collaborator

ljusten commented Jan 18, 2023

I hacked together a release build that contains the changes. I tested it to a local Docker container and it worked. Currently, I only have one Windows machine, so I can't test true remote copies, but I've applied for one.

https://github.com/google/cdc-file-transfer/suites/10439823585/artifacts/516488099

I'm super curious whether it works for you. Please let me know what issues you run into. Deploying the binaries properly is a surprisingly hard problem.

Note that we switched from scp to sftp, so you need to update your CDC_SCP_COMMAND to CDC_SFTP_COMMAND (but most likely you can just replace scp.exe by sftp.exe). Please see the readme contained in the zip.

@rstupek
Copy link

rstupek commented Jan 19, 2023

How do I get the copy to go on windows? Any hints on what command line to use would be appreciated

@ljusten
Copy link
Collaborator

ljusten commented Jan 30, 2023

Sorry for the delay, we had some distractions here lately. For local Windows syncing, i.e. copying files locally, you would just remove the user@host, e.g.

cdc_rsync C:\path\to\source\* C:\path\to\dest -rv

For remote Windows-to-Windows copies just add user@host:

cdc_rsync C:\path\to\source\* user@host:C:\path\to\dest -rv

The code will assume it's copying to Windows because the path starts with C: and deploy the right server file, i.e. cdc_rsync_server.exe.

@rstupek
Copy link

rstupek commented Jan 30, 2023

No worries on the delay, totally understand. I tried it and this is what I get:
Error: CreateProcess() failed: The system cannot find the file specified.; Failed to start cdc_rsync_server process; Failed to start server

I have the remove drive mapped so security wise the source server is logged into the destination server.

@LEOS2021
Copy link

LEOS2021 commented Oct 14, 2023

cdc-file-transfer-binaries-v1.0.0-x64
My code:

cdc_rsync D:\tmp\111\* D:\tmp\ggg -vr

Error: No remote host specified in destination 'D:\tmp\ggg'. Expected [user@]host:destination.

cdc-file-transfer-binaries-main-x64
My code:

cdc_rsync D:\tmp\111\* D:\tmp\ggg -vr

Port 51048: Server is listening
�EOF detected; Failed to receive packet of size 4; Failed to receive packet; Message pump thread is down; Failed to dequeue packet; Failed to receive SetOptionsRequest; Failed to receive options�2113 file(s) and 1949 folder(s) found
2532 file(s) and 2323 folder(s) found

@danildavletov
Copy link

danildavletov commented Nov 1, 2023

Error: No remote host specified in destination 'D:\tmp\ggg'.

I have the same issue. Any workarounds?

@graphixillusion
Copy link

Error: No remote host specified in destination 'D:\tmp\ggg'.

I have the same issue. Any workarounds?

Same for me. Same error using windows to windows local sync:

Port 51890: Server is listening
EOF detected; Failed to receive packet of size 4; Failed to receive packet; Message pump thread is down; Failed to dequeue packet; Failed to receive SetOptionsRequest; Failed to receive options582 file(s) and 21 folder(s) found

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

6 participants