What we know so far: libgit2/libgit2#4343 (comment)

Either we hope that these sums don't change too often, bite the bullet this time and wait for the next. Or we disable SHA sums for all github tarballs, and start moving all formulas for which there exists an alternative hosting to that.

We could also in theory start mirroring every github tarball, but that seems like a no-no to me (cf: Sisyphus).

We could also in theory start mirroring every github tarball, but that seems like a no-no to me (cf: Sisyphus).

That will not actually help, unless we made the mirror the primary, because we don't fall through to a mirror on a checksum mismatch.

Oh god. This is going to be actual hell.

Or we disable SHA sums for all github tarballs, and start moving all formulas for which there exists an alternative hosting to that.

Tossing any form of integrity verification out of the window seems like a drastic first step to take. Ideally, if anyone is extremely bored, projects need to get nagged to start doing actual release tarballs & have explained to them why this is important.

Tossing any form of integrity verification out of the window seems like a drastic first step to take.

I'm just going to assume @fxcoudert was being facetious. 🙉

😆 In my defence, with how busy this week is proving to be, if you tell me that you're actually a 🐐 I'd probably take it at face value for at least an hour or so before thinking "Wait a second".

I am actually a 🐐.

The team used to joke Xu was very possibly a non-malicious botnet because of how quickly high-quality solutions & ideas were churned out and implemented, so, rule nothing out I guess 🙈.

Either we hope that these sums don't change too often, bite the bullet this time and wait for the next.

Historically, they don't change that often, and certainly we (GitHub) don't plan to make gratuitous changes. We're discussing the possibility of providing byte-stable tarballs automatically (by cementing the archives, bugs and all, at the time a tag is uploaded).

In the meantime, what can we do to help resolve the pain of this particular change? If you provided a list of archive URLs, would it be helpful to get back a list of the updated hash for each case? That would allow a mass-update of the homebrew formulas.

We have 1498 URLs matching github.com/.*/archive/ in homebrew-core and homebrew-science), so I guess that's the extent of our potential problem.

But from where I'm sitting, not all of them are returning hashes that differ from the stored values. For example, https://github.com/01org/hyperscan/archive/v4.5.2.tar.gz still gives the value of 1f8fa44e94b642e54edc6a74cb8117d01984c0e661a15cad5a785e3ba28d18f5 we have on record, while https://github.com/airspy/host/archive/v1.0.9.tar.gz currently yields (for me) 967ef256596d4527b81f007f77b91caec3e9f5ab148a8fec436a703db85234cc which is different from our expectation of 358fea19f90bde13babc57ee7fdefeff3d8d8f5d629b0891734c5d4e811e8e6b.

To do a mass update we need things to have stabilised.

@peff See #18048

In the meantime, what can we do to help resolve the pain of this particular change?

Unfortunately, the major concern is whether the underlying files have changed, which we can only know

a) if we have access to the original tarballs to compare them to the new tarballs
b) if we have some way to reconstruct the old checksum from the current archives

In the meantime, what can we do to help resolve the pain of this particular change? If you provided a list of archive URLs, would it be helpful to get back a list of the updated hash for each case? That would allow a mass-update of the homebrew formulas.

The list is not fixed since I assume not all have been converted to the new format yet.

We're discussing the possibility of providing byte-stable tarballs automatically (by cementing the archives, bugs and all, at the time a tag is uploaded).

@peff please, please, please.... That would be wonderful ❤️

Not all hashes will change. Most of the changes are due to the bugfix in git/git@22f0dcd, so only tarballs with filenames greater than 100 characters are affected.

You're right that there may also be some caching in effect. We can flush those caches, which give us a stable state.

a) if we have access to the original tarballs to compare them to the new tarballs
b) if we have some way to reconstruct the old checksum from the current archives

You can verify the change by running a version of Git both with and without that patch (i.e., git revert 22f0dcd9634a818a0c83f23ea1a48f2d620c0546 and build the result). For instance:

$ git clone https://github.com/smira/aptly
$ cd aptly
$ /path/to/reverted/build/bin-wrappers/git archive --format=tar.gz --prefix=aptly-1.1.1/ v1.1.1 | sha256sum
92aa5caa12d756cb7469fa5772a03d7631b73d655b7329408a4d597ee8fb0ba4  -
$ /path/to/modern/git/bin-wrappers/git archive --format=tar.gz --prefix=aptly-1.1.1/ v1.1.1 | sha256sum
f7bc97e46cbff2f194af2c09db099d252f290a2ea90c251adea115e6d66cf31d  -

which matches what's in #18048. A few caveats, though:

• the prefix is obviously important, and comes from the canonical repo name. So if a repo is renamed, GitHub will issue a redirect from the old name, but the resulting tarball will have the new prefix in it. airspy/host is an example of this, and probably has been broken since it was renamed in March.

• note that the v is stripped when producing the prefix name from the tag name

• this is calling gzip -cn under the hood. Which is pretty stable, but it's possible your local gzip may produce slightly different output, wrecking the hash

It looks like git will use brew's version of gzip if it happens to be installed, and that produces the correct checksums.

iMac-TMP:aptly joe$ /usr/local/opt/git-reverted/bin/git archive --format=tar.gz --prefix=aptly-1.1.1/ v1.1.1 | shasum -a256
92aa5caa12d756cb7469fa5772a03d7631b73d655b7329408a4d597ee8fb0ba4  -
iMac-TMP:aptly joe$ /usr/local/opt/git/bin/git archive --format=tar.gz --prefix=aptly-1.1.1/ v1.1.1 | shasum -a256
f7bc97e46cbff2f194af2c09db099d252f290a2ea90c251adea115e6d66cf31d  -

So this looks promising.

For stable, I have verified all of the original checksums could be reconstructed except for

1. actual retags:

hh: dvorka/hstr#231
ndpi: ntop/nDPI#446
openvdb: https://github.com/dreamworksanimation/openvdb/issues/172
ssdb: ideawu/ssdb#1139

So far ssdb is the only upstream to acknowledge the retag.

2. kops, which is likely somehow related to the problem described in Was 1.6.0 retagged? kubernetes/kops#2630 and Github source code archives are not immutable (?) kubernetes/kubernetes#46443. I have informed upstream here: Was 1.6.0 retagged? kubernetes/kops#2630 (comment)

3. kubernetes-cli@1.3, which was already known to be wrong due to those same two issues (Was 1.6.0 retagged? kubernetes/kops#2630 and Github source code archives are not immutable (?) kubernetes/kubernetes#46443) as noted here: Github source code archives are not immutable (?) kubernetes/kubernetes#46443 (comment)

So I have merged #18048 and reverted (in #18053) the updates for hh, ndpi, and openvdb pending responses from upstream, and will not revert the others since they're accounted for.

On to devel next …

The devel specs for freerdp and hub (irony, much?) needed to be fixed:
#18055

I was able to reconstruct the original checksum for both.

The remaining mismatches will likely be in resource blocks.

Two checksums that were fine yesterday have gone bad today. I'm fixing them in #18072.

povray: https://github.com/POV-Ray/povray/archive/v3.7.0.3.tar.gz
teleport: https://github.com/gravitational/teleport/archive/v2.2.7.tar.gz

@ilovezfs I did a complete run over all of the GitHub archive URLs I could find in homebrew-core, and picked up a few stragglers. The results are in #18073, along with the scripts I used (they clone each repo, so unless you're running them on a fast connection, it will take a while).

I did a complete run

Note that this just looked for changes due to the Git patch. I didn't look for retags, renames, etc, that would cause the formula hashes to be invalid.

You lot are all amazing. Thanks for your fast and hard work to resolve this issue so quickly.

brew install hh

Error: SHA256 mismatch
Expected: c4995e7041dc66e2118f83bd4c6c7f4cff5b4c493ca28bd7e4aef76edeff71ba
Actual: 384fee04e4c80a1964dcf443131c1da4a20dd474fb48132a51d3de0a946ba996

@ilyaskorik yes that is an actual retag, which I've reported upstream, as noted here: #18044 (comment)

@ilovezfs But I can not install this program

@ilyaskorik yes, you'll need to wait for upstream to respond to dvorka/hstr#231 or risk changing the checksum yourself.

@peff Do you mind clarifying something?

Not all hashes will change. Most of the changes are due to the bugfix in git/git@22f0dcd, so only tarballs with filenames greater than 100 characters are affected.

With that in mind, why is the tagged 'release' https://github.com/airspy/host/archive/v1.0.9.tar.gz affected, the filename there is way shorter than 100 characters... Am I overlooking something?

In the long run, what is the best alternative here to avoid being hit from possible future changes in the generated tarballs?
Do a git clone (or download & unpack) of the tagged version and create a tarball yourself (via tar cfvz)? I guess that wouldn't be reliable either to generate a tarball that has the same checksum on different systems, for pretty much the same reasons that cause this issue in the first place...

With that in mind, why is the tagged 'release' https://github.com/airspy/host/archive/v1.0.9.tar.gz affected, the filename there is way shorter than 100 characters... Am I overlooking something?

@boegel The airspy/host hash was not affected by any change to the tarball-generating code. Its contents changed because the repository was renamed, and we use the canonical repository name as the tarball prefix. We provide an HTTP redirect from the old name, though you probably would want to update to the new URL along with the new hash. E.g.:

$ curl -w '%{redirect_url}' -o /dev/null https://github.com/airspy/host/archive/v1.0.9.tar.gz 
https://github.com/airspy/airspyone_host/archive/v1.0.9.tar.gz

In the long run, what is the best alternative here to avoid being hit from possible future changes in the generated tarballs?

Right now, I think there are basically two options:

1. Projects can create their own tarball using git archive (or tar, or whatever) and then upload them as an artifact to the Releases page. These are stored byte-for-byte indefinitely. Many projects already do this because they run autoconf or similar to generate their release tarballs. The big downsides are:

• The UI on the Releases page still lists the auto-generated tarball (albeit at the bottom), making it potentially confusing for visitors.

• This is an extra step that is borne by the project maintainer. But it's the packager who deals with the fallout from changing hashes. So the incentives are ,isaligned. And my understanding is that in general it's hard for packagers like homebrew to convince project maintainers to follow specific workflows.

2. Verification doesn't necessarily have to be over the byte-for-byte tarball contents. It could be over some canonicalized representation (e.g., concatenating the filenames and contents in order and taking the sha256sum of that). The problems there are:

• A convenient tool doesn't already exist (though it would not be too hard to come up with one).

• It increases the attack surface for impersonating an official tarball, since an attacker has more leeway to change the bytes in a way that keeps their tarball matching the canonical hash, but which may behave differently in practice.

• Likewise, it increases the attack surface of the overall system. Right now nothing gets fed to tar until its hash is checked. If your canonical-hash-checker uses tar under the hood, then an attacker may target bugs in tar even before the hash is checked. You can say the same for existing hash checkers, of course, but it's a lot more likely to have bugs in something as complicated as tar versus sha256sum.

  I don't know if those are compelling downsides or not.

That's a bit of a brain dump on the subject. There may be other options that I haven't thought of, too.

We're discussing the possibility of providing byte-stable tarballs automatically (by cementing the archives, bugs and all, at the time a tag is uploaded).

@peff is this still in the cards as a possibility?

@ilovezfs Sorry, I don't have an update. It's not something I'd be working on personally, but my understanding is it's still being explored.

@peff OK, thanks :)

@peff Thanks for the extensive reply, much appreciated!

@peff: Thanks from me for the detailed response, as well.

I think ensuring byte-stable tarballs, either by saving them or using something like Debian's special tar arguments, is the way to go here. We were hit with this in Spack as well, and I'm having trouble deciding exactly how to resolve this, since it's ambiguous whether there are going to be any guarantees on tarballs in /archive/ links going forward.

If you do not end up guaranteeing anything about /archive/ links, I'll probably convert our 7-800 affected packages to use /release where possible, and use git clone for specific commits where there is no stable release. I may also try to convince some package maintainers to post actual releases, but I agree strongly with all the downsides to that which you mentioned, particularly:

my understanding is that in general it's hard for packagers like homebrew to convince project maintainers to follow specific workflows.

It would be good if you put a warning on the release page about the /archive/ links so that other folks aren't encouraged to rely on their hashes.