Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PreComp crashed on big file (ERROR: 3) #65

Closed
pasha-zzz opened this issue Sep 22, 2017 · 7 comments
Closed

PreComp crashed on big file (ERROR: 3) #65

pasha-zzz opened this issue Sep 22, 2017 · 7 comments
Assignees
Labels

Comments

@pasha-zzz
Copy link

PreComp 0.46 (64 bit), Win 10 x64.

Files in dir after crash (file starting with D5803 - source):
~temp00000000.dat (0 B)
~temp000000000.dat (16382 B)
~temp000000001.dat (1735 MB)
~temp000000001.dat_ (2241 MB)
D5803 23.5.A.1.291 Customized RU 1289-0445 R3D (by skapunkcsd90@4pda).ftf (1616 MB)
precomp.exe (1515 KB)

Command line:
precomp.exe -cn -intense -brute "D5803 23.5.A.1.291 Customized RU 1289-0445 R3D (by skapunkcsd90@4pda).ftf"
(and with compression, -cl, - same thing)

94.04% -
ERROR 3: There is not enough space on disk

Even other PC and OS - same position (94.04%) and same error 3. Other similar FTF files >1GB size - same issue (other pos). But for 700 MB file works fine.
Path to folder - C:\PreComp. Disk C is NTFS, 652 GB free.

@schnaader
Copy link
Owner

The temporary file with an underscore is recursion input, so recursion might have problems with streams >2 GB. There are several things you can try:

  • Run with "-cn -intense -brute -d0" - might resolve the error if it is due to recursion.
  • Run with "-cn -intense -brute -v" - this will enable verbose mode so you should see more information before the error occurs. If you do this, you can post the last messages here, it might help me fixing the bug.
  • Upload one of the files where the error occurs somewhere so I can reproduce the problem.

@schnaader schnaader self-assigned this Sep 22, 2017
@schnaader schnaader added the bug label Sep 22, 2017
@schnaader schnaader added this to the Precomp v0.4.7 milestone Sep 22, 2017
@pasha-zzz
Copy link
Author

This file - https://cloud.mail.ru/public/Ju55/8e5Z3UZGo

Log with -v:
`(82.37%) Possible zLib-Stream in multiPNG found at position 1818935853, windowbits = 15
Compressed size: 188859
Can be decompressed to 1920800 bytes
No matches
ZIP header detected
(82.38%) ZIP header detected at position 1819125014
compressed size: 3284
uncompressed size: 3284
file name length: 32
extra field length: 0
(82.38%) Possible zLib-Stream in PNG found at position 1819125113, windowbits = 14
Compressed size: 3221
Can be decompressed to 12352 bytes
No matches
ZIP header detected
(82.38%) ZIP header detected at position 1819128360
compressed size: 5993
uncompressed size: 5993
file name length: 31
extra field length: 3
(82.38%) Possible zLib-Stream in PNG found at position 1819128461, windowbits = 14
Compressed size: 5930
Can be decompressed to 12352 bytes
No matches
ZIP header detected
(82.38%) ZIP header detected at position 1819134417
compressed size: 594
uncompressed size: 594
file name length: 36
extra field length: 1
(82.38%) Possible zLib-Stream in PNG found at position 1819134890, windowbits = 9
Compressed size: 162
Can be decompressed to 272 bytes
Identical recompressed bytes: 162 of 162
Identical decompressed bytes: 272 of 272
Best match with level combination 98: 162 bytes, decompressed to 272 bytes
ZIP header detected
(82.38%) ZIP header detected at position 1819135078
compressed size: 4303
uncompressed size: 4303
file name length: 37
extra field length: 3
(82.38%) Possible zLib-Stream in PNG found at position 1819135185, windowbits = 14
Compressed size: 4240
Can be decompressed to 12352 bytes
No matches
ZIP header detected
(82.38%) ZIP header detected at position 1819139451
compressed size: 30716
uncompressed size: 30716
file name length: 41
extra field length: 2
(82.38%) Possible zLib-Stream in multiPNG found at position 1819139561, windowbits = 15
Compressed size: 30617
Can be decompressed to 97380 bytes
No matches
ZIP header detected
(82.38%) ZIP header detected at position 1819170240
compressed size: 10650
uncompressed size: 10650
file name length: 30
extra field length: 0
(82.38%) Possible zLib-Stream in multiPNG found at position 1819170337, windowbits = 15
Compressed size: 10575
Can be decompressed to 36960 bytes
No matches
ZIP header detected
(82.38%) ZIP header detected at position 1819180950
compressed size: 257
uncompressed size: 520
file name length: 19
extra field length: 0
(82.38%) Possible zLib-Stream in ZIP found at position 1819180950
Compressed size: 257
Can be decompressed to 520 bytes
Identical recompressed bytes: 257 of 257
Identical decompressed bytes: 520 of 520
Best match with level combination 98, windowbits = 15: 257 bytes, decompressed to 520 bytes
Recursion start - new recursion depth 2
No recursion streams found
Recursion end - back to recursion depth 1
ZIP header detected
(82.38%) ZIP header detected at position 1819181256
compressed size: 57
uncompressed size: 69
file name length: 22
extra field length: 0
(82.38%) Possible zLib-Stream in ZIP found at position 1819181256
Compressed size: 57
Can be decompressed to 69 bytes
Identical recompressed bytes: 57 of 57
Identical decompressed bytes: 69 of 69
Best match with level combination 98, windowbits = 15: 57 bytes, decompressed to 69 bytes
Recursion start - new recursion depth 2
No recursion streams found
Recursion end - back to recursion depth 1
ZIP header detected
(82.38%) ZIP header detected at position 1819181365
compressed size: 250
uncompressed size: 537
file name length: 22
extra field length: 0
(82.38%) Possible zLib-Stream in ZIP found at position 1819181365
Compressed size: 250
Can be decompressed to 537 bytes
Identical recompressed bytes: 250 of 250
Identical decompressed bytes: 537 of 537
Best match with level combination 98, windowbits = 15: 250 bytes, decompressed to 537 bytes
Recursion start - new recursion depth 2
No recursion streams found
Recursion end - back to recursion depth 1
ZIP header detected
(82.38%) ZIP header detected at position 1819181667
compressed size: 4500
uncompressed size: 5284
file name length: 17
extra field length: 0
(82.38%) Possible zLib-Stream in ZIP found at position 1819181667
Compressed size: 4500
Can be decompressed to 5284 bytes
Identical recompressed bytes: 4500 of 4500
Identical decompressed bytes: 5284 of 5284
Best match with level combination 98, windowbits = 15: 4500 bytes, decompressed to 5284 bytes
Recursion start - new recursion depth 2
No recursion streams found
Recursion end - back to recursion depth 1
ZIP header detected
(82.38%) ZIP header detected at position 1819186214
compressed size: 109
uncompressed size: 156
file name length: 26
extra field length: 0
(82.38%) Possible zLib-Stream in ZIP found at position 1819186214
Compressed size: 109
Can be decompressed to 156 bytes
Identical recompressed bytes: 109 of 109
Identical decompressed bytes: 156 of 156
Best match with level combination 98, windowbits = 15: 109 bytes, decompressed to 156 bytes
Recursion start - new recursion depth 2
No recursion streams found
Recursion end - back to recursion depth 1
ZIP header detected
(82.38%) ZIP header detected at position 1819186379
compressed size: 267
uncompressed size: 589
file name length: 26
extra field length: 0
(82.38%) Possible zLib-Stream in ZIP found at position 1819186379
Compressed size: 267
Can be decompressed to 589 bytes
Identical recompressed bytes: 267 of 267
Identical decompressed bytes: 589 of 589
Best match with level combination 98, windowbits = 15: 267 bytes, decompressed to 589 bytes
Recursion start - new recursion depth 2
No recursion streams found
Recursion end - back to recursion depth 1
ZIP header detected
(82.38%) ZIP header detected at position 1819186702
compressed size: 124
uncompressed size: 206
file name length: 26
extra field length: 0
(82.38%) Possible zLib-Stream in ZIP found at position 1819186702
Compressed size: 124
Can be decompressed to 206 bytes
Identical recompressed bytes: 124 of 124
Identical decompressed bytes: 206 of 206
Best match with level combination 98, windowbits = 15: 124 bytes, decompressed to 206 bytes
Recursion start - new recursion depth 2
No recursion streams found
Recursion end - back to recursion depth 1
ZIP header detected
(82.38%) ZIP header detected at position 1819186882
compressed size: 143
uncompressed size: 277
file name length: 26
extra field length: 0
(82.38%) Possible zLib-Stream in ZIP found at position 1819186882
Compressed size: 143
Can be decompressed to 277 bytes
Identical recompressed bytes: 143 of 143
Identical decompressed bytes: 277 of 277
Best match with level combination 98, windowbits = 15: 143 bytes, decompressed to 277 bytes
Recursion start - new recursion depth 2
No recursion streams found
Recursion end - back to recursion depth 1
ZIP header detected
(82.38%) ZIP header detected at position 1819187081
compressed size: 189
uncompressed size: 316
file name length: 21
extra field length: 0
(82.38%) Possible zLib-Stream in ZIP found at position 1819187081
Compressed size: 189
Can be decompressed to 316 bytes
Identical recompressed bytes: 189 of 189
Identical decompressed bytes: 316 of 316
Best match with level combination 98, windowbits = 15: 189 bytes, decompressed to 316 bytes
Recursion start - new recursion depth 2
No recursion streams found
Recursion end - back to recursion depth 1
ZIP header detected
(82.38%) ZIP header detected at position 1819187321
compressed size: 10912
uncompressed size: 10912
file name length: 14
extra field length: 3
ZIP header detected
(82.38%) ZIP header detected at position 1819198280
compressed size: 669
uncompressed size: 1481
file name length: 20
extra field length: 0
(82.38%) Possible zLib-Stream in ZIP found at position 1819198280
Compressed size: 669
Can be decompressed to 1481 bytes
Identical recompressed bytes: 669 of 669
Identical decompressed bytes: 1481 of 1481
Best match with level combination 68, windowbits = 15: 669 bytes, decompressed to 1481 bytes
Recursion start - new recursion depth 2
No recursion streams found
Recursion end - back to recursion depth 1
ZIP header detected
(82.38%) ZIP header detected at position 1819198999
compressed size: 697
uncompressed size: 1534
file name length: 54
extra field length: 0
(82.38%) Possible zLib-Stream in ZIP found at position 1819198999
Compressed size: 697
Can be decompressed to 1534 bytes
Identical recompressed bytes: 697 of 697
Identical decompressed bytes: 1534 of 1534
Best match with level combination 68, windowbits = 15: 697 bytes, decompressed to 1534 bytes
Recursion start - new recursion depth 2
No recursion streams found
Recursion end - back to recursion depth 1
ZIP header detected
(82.38%) ZIP header detected at position 1819199780
compressed size: 1138
uncompressed size: 1373
file name length: 55
extra field length: 0
(82.38%) Possible zLib-Stream in ZIP found at position 1819199780
Compressed size: 1138
Can be decompressed to 1373 bytes
Identical recompressed bytes: 1138 of 1138
Identical decompressed bytes: 1373 of 1373
Best match with level combination 98, windowbits = 15: 1138 bytes, decompressed to 1373 bytes
Recursion start - new recursion depth 2
No recursion streams found
Recursion end - back to recursion depth 1
Recursion streams found
Recursion end - back to recursion depth 0

ERROR 3: There is not enough space on disk`

@pasha-zzz
Copy link
Author

But if I use -d0 - precomp poduces smaller (more compressed) file...

@schnaader
Copy link
Owner

Yes, using -d0 is just one possible workaround to get the file processed until the bug is fixed. Thanks for the feedback and the file, I'm looking into it.

@pasha-zzz
Copy link
Author

Yes, -d0 helps.

schnaader added a commit that referenced this issue Oct 17, 2017
- Use 64 bit for byte count
- Remove unused parameter
@schnaader
Copy link
Owner

The crash (disk full error) is fixed with the commit above (22e5f2b), but I'll leave the issue open to fix another thing that I recognized:

Brute mode detects a part of the big stream inside the ZIP, but it should already be handled by the ZIP parser without brute mode. None of the zlib parsers, however, does process streams above 2 GB correctly. So the crash could be fixed, but fixing the other zlib parser issues will improve compression ratio for such files and brute mode won't be necessary here anymore.

@schnaader
Copy link
Owner

Completely fixed with the commit series 24d4378 ... 27053a4. Now, even without brute or even intense mode, the ZIP stream is recognized and processed:

Description resulting file size in bytes (-cn) resulting file size in bytes (-cl)
Before 1,945,185,430 1,639,140,905
Before, -intense -brute 2,965,380,085 1,471,676,589
After 3,385,832,046 1,417,452,005

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants