Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

clean up and export crc32c function #22274

Merged
merged 14 commits into from
Jun 13, 2017
Merged

clean up and export crc32c function #22274

Changes from 1 commit
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
643b1e3
clean up and export crc32c function
stevengj Jun 7, 2017
721bbf5
added PR to NEWS
stevengj Jun 7, 2017
184acd1
use read, not mmap, for crc
stevengj Jun 7, 2017
c3f5d91
slight optimization (don't open cachefile twice for checksum)
stevengj Jun 7, 2017
bd64651
avoid resizing buffer in readbytes
stevengj Jun 7, 2017
62d7dc4
add an assertion
stevengj Jun 7, 2017
ceda720
more crc32c tests
stevengj Jun 7, 2017
67ef143
docs typo
stevengj Jun 8, 2017
f34d08c
crc32c(IOBuffer) bugfix and test
stevengj Jun 8, 2017
38f6fc0
restore crc32 of String, add crc32c(io) to read all of a stream, add …
stevengj Jun 8, 2017
1126d0f
use crc32c block size of 8192*3, matching the underling C library
stevengj Jun 9, 2017
da2fba1
optimized IOBuffer crc32c
stevengj Jun 9, 2017
a1b0431
slight simplification/generalization
stevengj Jun 12, 2017
8e058f3
eliminate theoretical race condition in crc32c if file is being writt…
stevengj Jun 12, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
use crc32c block size of 8192*3, matching the underling C library
  • Loading branch information
@stevengj
stevengj committed Jun 9, 2017
commit 1126d0ffa6c285a33ce04d8439246c1ba21dce38
6 changes: 4 additions & 2 deletions base/util.jl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -804,8 +804,10 @@ mixed with a starting `crc` integer. If `nb` is not supplied, then
"""
function crc32c(io::IO, nb::Integer, crc::UInt32=0x00000000)
nb < 0 && throw(ArgumentError("number of bytes to checksum must be ≥ 0"))
buf = Array{UInt8}(min(nb, 16384))
while !eof(io) && nb > 16384
# use block size 24576=8192*3, since that is the threshold for
# 3-way parallel SIMD code in the underlying jl_crc32c C function.
buf = Array{UInt8}(min(nb, 24576))
while !eof(io) && nb > 24576
n = readbytes!(io, buf)
crc = unsafe_crc32c(buf, n, crc)
nb -= n
Expand Down