Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support hard links in directories (Trac #63) #92

Open
boxbackup-bot opened this issue May 18, 2019 · 5 comments
Open

Support hard links in directories (Trac #63) #92

boxbackup-bot opened this issue May 18, 2019 · 5 comments

Comments

@boxbackup-bot
Copy link

boxbackup-bot commented May 18, 2019

Ben Bennett reported the following warning message:

Jun 21 22:23:42 ayup bbackupd[24222]: WARNING: Found conflicting parent ID for file ID 1579976 (/home/fiji/public_html/pictures/49/014 - Nursery.jpg): expected 34689 but found 69964 (same directory used in two different locations?)

The warning is correct, the file is in two locations...

# ls -li '/home/fiji/public_html/pictures/49/013 - Nursery.jpg' '/home/fiji/public_html/family/3/013 - Nursery.jpg'
1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/family/3/013 - Nursery.jpg
1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/pictures/49/013 - Nursery.jpg

Chris Wilson said:

I agree that hard links should be supported, and that this is a
bug. Please could you get an account in the bug tracker and file
a bug?

The reasons for this warning are:

  1. Almost nobody uses hard links on their data files, so nobody
    has reported this problem before.
  1. Including the same files in two backup locations (not two
    directories) is an error. Backing up the same file data twice
    would be a waste of bandwidth.
  1. There was a bug on Windows where file IDs were being
    duplicated all the time, which was only spotted because I added
    this warning.
  1. Box Backup uses file IDs to track renaming of files. If you
    use hard links then this will not work correctly and will cause
    your files to be renamed between the two directories all the
    time. This is the real bug.

Stefan Müller Wildi said:

I suggest introducing options to either suppress such warning
messages altogether (maybe by type if that is possible?) or to
reduce the number of warnings of identical type to 1 (like
"showing only one of ##"). Something similar must already be
implemented for I also read:

"c314-stefan bbackupd[1574]: WARNING: Suppressing duplicate
notification about read-error"

'''This bug can cause data corruption in your backups! Do not ignore it! The warning is a warning for a reason! '''

Migrated from https://www.boxbackup.org/ticket/63

{
    "status": "assigned", 
    "changetime": "2015-01-02T22:21:55", 
    "description": "Ben Bennett reported the following warning message:\n\nJun 21 22:23:42 ayup bbackupd[24222]: WARNING: Found conflicting parent ID for file ID 1579976 (/home/fiji/public_html/pictures/49/014 - Nursery.jpg): expected 34689 but found 69964 (same directory used in two different locations?)\n\nThe warning is correct, the file is in two locations...\n\n{{{\n# ls -li '/home/fiji/public_html/pictures/49/013 - Nursery.jpg' '/home/fiji/public_html/family/3/013 - Nursery.jpg'\n1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/family/3/013 - Nursery.jpg\n1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/pictures/49/013 - Nursery.jpg\n}}}\n\nChris Wilson said:\n\n> I agree that hard links should be supported, and that this is a \n> bug. Please could you get an account in the bug tracker and file\n> a bug?\n\n> The reasons for this warning are:\n\n> 1. Almost nobody uses hard links on their data files, so nobody \n> has reported this problem before.\n \n> 2. Including the same files in two backup locations (not two \n> directories) is an error. Backing up the same file data twice \n> would be a waste of bandwidth.\n\n> 3. There was a bug on Windows where file IDs were being \n> duplicated all the time, which was only spotted because I added \n> this warning.\n \n> 4. Box Backup uses file IDs to track renaming of files. If you \n> use hard links then this will not work correctly and will cause \n> your files to be renamed between the two directories all the \n> time. This is the real bug.\n\nStefan M\u00fcller Wildi said:\n\n> I suggest introducing options to either suppress such warning \n> messages altogether (maybe by type if that is possible?) or to \n> reduce the number of warnings of identical type to 1 (like \n> \"showing only one of ##\"). Something similar must already be \n> implemented for I also read:\n\n> \"c314-stefan bbackupd[1574]: WARNING: Suppressing duplicate \n> notification about read-error\"\n\n'''This bug can cause data corruption in your backups! Do not ignore it! The warning is a warning for a reason! '''", 
    "reporter": "bbennett", 
    "cc": "sm8ps-boxbackup1@yahoo.com", 
    "resolution": "", 
    "_ts": "1420237315253075", 
    "component": "bbackupd", 
    "summary": "Support hard links in directories", 
    "priority": "normal", 
    "keywords": "backup duplicate rename hardlink corruption", 
    "version": "0.11rc2", 
    "time": "2010-01-11T04:34:03", 
    "milestone": "", 
    "owner": "chris", 
    "type": "defect"
}
@boxbackup-bot
Copy link
Author

Trac update at 20100331T07:44:42:

  • chris changed keywords from "" to "backup duplicate rename hardlink corruption"
  • chris changed cc from "" to "sm8ps-boxbackup1@yahoo.com"
  • chris changed description from:

I get:
Jun 21 22:23:42 ayup bbackupd[24222]: WARNING: Found conflicting parent ID for file ID 1579976 (/home/fiji/public_html/pictures/49/014 -
Nursery.jpg): expected 34689 but found 69964 (same directory used in two different locations?)

The warning is correct, the file is in two locations...

ls -li '/home/fiji/public_html/pictures/49/013 - Nursery.jpg' '/home/fiji/public_html/family/3/013 - Nursery.jpg'
1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/family/3/013 - Nursery.jpg
1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/pictures/49/013 - Nursery.jpg

Chris Wilson said:
I agree that hard links should be supported, and that this is a bug.
Please could you get an account in the bug tracker and file a bug?

The reasons for this warning are:

  1. Almost nobody uses hard links on their data files, so nobody has
    reported this problem before.

  2. Including the same files in two backup locations (not two directories)
    is an error and a waste of bandwidth.

  3. There was a bug on Windows where file IDs were being duplicated all the
    time, which was only spotted because I added this warning.

  4. Box Backup uses file IDs to track renaming of files. If you use hard
    links then this will not work correctly and will cause your files to be
    renamed between the two directories all the time. This is the real bug.

to:

I get:
Jun 21 22:23:42 ayup bbackupd[24222]: WARNING: Found conflicting parent ID for file ID 1579976 (/home/fiji/public_html/pictures/49/014 -
Nursery.jpg): expected 34689 but found 69964 (same directory used in two different locations?)

The warning is correct, the file is in two locations...

{{{

ls -li '/home/fiji/public_html/pictures/49/013 - Nursery.jpg' '/home/fiji/public_html/family/3/013 - Nursery.jpg'

1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/family/3/013 - Nursery.jpg
1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/pictures/49/013 - Nursery.jpg
}}}

Chris Wilson said:

I agree that hard links should be supported, and that this is a
bug. Please could you get an account in the bug tracker and file
a bug?

The reasons for this warning are:

  1. Almost nobody uses hard links on their data files, so nobody
    has reported this problem before.

  2. Including the same files in two backup locations (not two
    directories) is an error. Backing up the same file data twice
    would be a waste of bandwidth.

  3. There was a bug on Windows where file IDs were being
    duplicated all the time, which was only spotted because I added
    this warning.

  4. Box Backup uses file IDs to track renaming of files. If you
    use hard links then this will not work correctly and will cause
    your files to be renamed between the two directories all the
    time. This is the real bug.

Stefan Müller Wildi said:

I suggest introducing options to either suppress such warning
messages altogether (maybe by type if that is possible?) or to
reduce the number of warnings of identical type to 1 (like
"showing only one of ##"). Something similar must already be
implemented for I also read:

"c314-stefan bbackupd[1574]: WARNING: Suppressing duplicate
notification about read-error"

'''This bug can cause data corruption in your backups! Do not ignore it! The warning is a warning for a reason!'''

@boxbackup-bot
Copy link
Author

Trac update at 20100331T07:45:44: chris changed description from:

I get:
Jun 21 22:23:42 ayup bbackupd[24222]: WARNING: Found conflicting parent ID for file ID 1579976 (/home/fiji/public_html/pictures/49/014 -
Nursery.jpg): expected 34689 but found 69964 (same directory used in two different locations?)

The warning is correct, the file is in two locations...

{{{

ls -li '/home/fiji/public_html/pictures/49/013 - Nursery.jpg' '/home/fiji/public_html/family/3/013 - Nursery.jpg'

1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/family/3/013 - Nursery.jpg
1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/pictures/49/013 - Nursery.jpg
}}}

Chris Wilson said:

I agree that hard links should be supported, and that this is a
bug. Please could you get an account in the bug tracker and file
a bug?

The reasons for this warning are:

  1. Almost nobody uses hard links on their data files, so nobody
    has reported this problem before.

  2. Including the same files in two backup locations (not two
    directories) is an error. Backing up the same file data twice
    would be a waste of bandwidth.

  3. There was a bug on Windows where file IDs were being
    duplicated all the time, which was only spotted because I added
    this warning.

  4. Box Backup uses file IDs to track renaming of files. If you
    use hard links then this will not work correctly and will cause
    your files to be renamed between the two directories all the
    time. This is the real bug.

Stefan Müller Wildi said:

I suggest introducing options to either suppress such warning
messages altogether (maybe by type if that is possible?) or to
reduce the number of warnings of identical type to 1 (like
"showing only one of ##"). Something similar must already be
implemented for I also read:

"c314-stefan bbackupd[1574]: WARNING: Suppressing duplicate
notification about read-error"

'''This bug can cause data corruption in your backups! Do not ignore it! The warning is a warning for a reason!'''

to:

I get:
Jun 21 22:23:42 ayup bbackupd[24222]: WARNING: Found conflicting parent ID for file ID 1579976 (/home/fiji/public_html/pictures/49/014 -
Nursery.jpg): expected 34689 but found 69964 (same directory used in two different locations?)

The warning is correct, the file is in two locations...

{{{

ls -li '/home/fiji/public_html/pictures/49/013 - Nursery.jpg' '/home/fiji/public_html/family/3/013 - Nursery.jpg'

1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/family/3/013 - Nursery.jpg
1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/pictures/49/013 - Nursery.jpg
}}}

Chris Wilson said:

I agree that hard links should be supported, and that this is a
bug. Please could you get an account in the bug tracker and file
a bug?

The reasons for this warning are:

  1. Almost nobody uses hard links on their data files, so nobody
    has reported this problem before.
  1. Including the same files in two backup locations (not two
    directories) is an error. Backing up the same file data twice
    would be a waste of bandwidth.
  1. There was a bug on Windows where file IDs were being
    duplicated all the time, which was only spotted because I added
    this warning.
  1. Box Backup uses file IDs to track renaming of files. If you
    use hard links then this will not work correctly and will cause
    your files to be renamed between the two directories all the
    time. This is the real bug.

Stefan Müller Wildi said:

I suggest introducing options to either suppress such warning
messages altogether (maybe by type if that is possible?) or to
reduce the number of warnings of identical type to 1 (like
"showing only one of ##"). Something similar must already be
implemented for I also read:

"c314-stefan bbackupd[1574]: WARNING: Suppressing duplicate
notification about read-error"

'''This bug can cause data corruption in your backups! Do not ignore it! The warning is a warning for a reason! '''

@boxbackup-bot
Copy link
Author

Trac update at 20100331T07:46:52:

  • chris changed description from:

I get:
Jun 21 22:23:42 ayup bbackupd[24222]: WARNING: Found conflicting parent ID for file ID 1579976 (/home/fiji/public_html/pictures/49/014 -
Nursery.jpg): expected 34689 but found 69964 (same directory used in two different locations?)

The warning is correct, the file is in two locations...

{{{

ls -li '/home/fiji/public_html/pictures/49/013 - Nursery.jpg' '/home/fiji/public_html/family/3/013 - Nursery.jpg'

1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/family/3/013 - Nursery.jpg
1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/pictures/49/013 - Nursery.jpg
}}}

Chris Wilson said:

I agree that hard links should be supported, and that this is a
bug. Please could you get an account in the bug tracker and file
a bug?

The reasons for this warning are:

  1. Almost nobody uses hard links on their data files, so nobody
    has reported this problem before.
  1. Including the same files in two backup locations (not two
    directories) is an error. Backing up the same file data twice
    would be a waste of bandwidth.
  1. There was a bug on Windows where file IDs were being
    duplicated all the time, which was only spotted because I added
    this warning.
  1. Box Backup uses file IDs to track renaming of files. If you
    use hard links then this will not work correctly and will cause
    your files to be renamed between the two directories all the
    time. This is the real bug.

Stefan Müller Wildi said:

I suggest introducing options to either suppress such warning
messages altogether (maybe by type if that is possible?) or to
reduce the number of warnings of identical type to 1 (like
"showing only one of ##"). Something similar must already be
implemented for I also read:

"c314-stefan bbackupd[1574]: WARNING: Suppressing duplicate
notification about read-error"

'''This bug can cause data corruption in your backups! Do not ignore it! The warning is a warning for a reason! '''

to:

Ben Bennett reported the following warning message:

Jun 21 22:23:42 ayup bbackupd[24222]: WARNING: Found conflicting parent ID for file ID 1579976 (/home/fiji/public_html/pictures/49/014 - Nursery.jpg): expected 34689 but found 69964 (same directory used in two different locations?)

The warning is correct, the file is in two locations...

{{{

ls -li '/home/fiji/public_html/pictures/49/013 - Nursery.jpg' '/home/fiji/public_html/family/3/013 - Nursery.jpg'

1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/family/3/013 - Nursery.jpg
1579975 -rw-r--r-- 2 fiji fiji 131622 2002-12-18 07:37 /home/fiji/public_html/pictures/49/013 - Nursery.jpg
}}}

Chris Wilson said:

I agree that hard links should be supported, and that this is a
bug. Please could you get an account in the bug tracker and file
a bug?

The reasons for this warning are:

  1. Almost nobody uses hard links on their data files, so nobody
    has reported this problem before.
  1. Including the same files in two backup locations (not two
    directories) is an error. Backing up the same file data twice
    would be a waste of bandwidth.
  1. There was a bug on Windows where file IDs were being
    duplicated all the time, which was only spotted because I added
    this warning.
  1. Box Backup uses file IDs to track renaming of files. If you
    use hard links then this will not work correctly and will cause
    your files to be renamed between the two directories all the
    time. This is the real bug.

Stefan Müller Wildi said:

I suggest introducing options to either suppress such warning
messages altogether (maybe by type if that is possible?) or to
reduce the number of warnings of identical type to 1 (like
"showing only one of ##"). Something similar must already be
implemented for I also read:

"c314-stefan bbackupd[1574]: WARNING: Suppressing duplicate
notification about read-error"

'''This bug can cause data corruption in your backups! Do not ignore it! The warning is a warning for a reason! '''

  • chris changed owner from "ben" to "chris"
  • chris changed status from "new" to "assigned"

@boxbackup-bot
Copy link
Author

Trac update at 20150102T22:21:55: chris commented:

Implementing this properly requires the long-awaited (and not yet finished) snapshot support to be completed first.

In the mean time, we need a test that hardlinked files are backed up properly (but independently) without warnings and without causing rename tracking to do bad things.

@qris
Copy link
Contributor

qris commented May 18, 2019

Commit 7970266 (on the s3_support branch, see #9) contains partial support for hard links. At least, the backup is no longer corrupted. But they are still backed up multiple times, wasting space on the backup server/client account, and will be duplicated instead of hardlinked together on restore. The solution still requires snapshot support, since this allows multiple directory entries to reference the same data object on the store.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants