Reduce the number of high-priority CVEs in scans #405
Conversation
acdha
force-pushed
the
patch-1
branch
2 times, most recently
from
0fc1734 to
64a9067
Compare
There was a problem hiding this comment.
Removing htrace will lead to ClassNotFoundException for anyone using Hdfs stuff. Moving Hadoop stuff to a module (aka "contrib") is happening in 9.x and that will help somewhat. We can't remove that in 8x's Docker image (sorry). I could see replacing the JAR if there is something suitable.
Modifying the Dockerfile directly is not how this project works. See Dockerfile-varsolr.template and we run scripts to regenerate. Also, removing stuff in the big "RUN" command would be superior as it saves space and a layer.
This updates the work @janhoy did in 0b13c14 by adjusting to a change in shell behaviour: now that dash is the default shell, the brace expansion does not work as expected but the `-f` flag passed to `rm` meant that the build continued without deleting the intended files. This partially addresses https://issues.apache.org/jira/browse/SOLR-15855 because the removed files have a number of open CVEs, several of which are serious and may block deployments if used in a CI/CD pipeline which has a scanner. Because dash does not support brace expansion and find’s `-delete` action does not support deleting directories, this commit splits the work into two commands: `rm -R` for the directories and `find … -delete` for the files.
|
This turned out to be more interesting than I expected: I noticed that my removal of the test-framework appeared to be completely unnecessary because @janhoy already did this 2 years ago in 0b13c14: 0b13c14#diff-8d7a21b017921bb88eaf71656b7b5767203db16e8126fc1e5ad2a9ba0bc542f5 Unfortunately, that no longer works as intended because the default shell is currently I'm guessing that when @janhoy made that change originally the default shell was bash and since everything targeted was by definition not a dependency it wasn't obvious that this no longer worked when the default shell changed. I removed the To work around the lack of brace expansion, I split that into two parts: removing the directories with - rm -Rf /opt/solr/docs/ /opt/solr/dist/{solr-core-$SOLR_VERSION.jar,solr-solrj-$SOLR_VERSION.jar,solrj-lib,solr-test-framework-$SOLR_VERSION.jar,test-framework}; \
+ rm -R /opt/solr/docs/ /opt/solr/dist/solrj-lib /opt/solr/dist/test-framework; \
+ find /opt/solr/dist/ \( -name "solr-core-$SOLR_VERSION.jar" -o -name "solr-solrj-$SOLR_VERSION.jar" -o -name "solr-test-framework-$SOLR_VERSION.jar" \) -delete; \ |
|
The HTrace issue has a potential fix thanks to https://issues.apache.org/jira/browse/HBASE-24802 but I wanted to check your course on this. The HBase project produced hbase-noop-htrace, which appears to be a drop-in replacement. Would it be considered cleaner to drop that in in the Docker build process or try to get that merged into the upstream Solr builds? |
|
Awesome investigation Chris! RE htrace: this should be done upstream (in Solr proper). Forewarning: Solr 9 (not yet released) has a very different build/dependency management than Solr 8 so sadly the work on each branch there is different but not hard in any case. You don't "have" to do it on both but if you do it on 8, it's kinda necessary to do so in 9 as well. I can help. There are tests for Hadoop related functionality, and so replacing it there gives some comfort we didn't break something. p.s. I'm on vacation this week in DC (I see you work there) so I'm not particularly responsive. |
|
Oh another thing: In 9x there's big effort to put hadoop stuff into another module so I suggest you not bother with the 9x side, leaving that side to me and/or the person doing that as a minor change. |
Please do not feel any pressure from me to do anything otherwise — I have deep sympathy for other maintainers.
That's good to know, thanks. I was wondering how big a project that was going to be since I think continuous security monitoring is going to increase the overhead of public services like this and was wondering whether it might make sense to have something like separate container tags for major features like Hadoop, but that's hardly free. |
|
I'd definitely be in favor of splitting the docker distro into two -- one is just "core" (+ prometheus exporter perhaps) and the other has all the so-called "contribs" in it (or perhaps drop that entirely). This is the kind of thing we can do at a 9.0 release. |
|
This may be an issue also for the 9.0 dockerfile template at https://github.com/apache/solr/blob/main/solr/docker/templates/Dockerfile.body.template - have not tested, but it likely has dash as default shell too... |
|
Note that once solr-core.jar is successfully deleted, the |
This has two of the changes discussed in https://issues.apache.org/jira/browse/SOLR-15855 which will reduce the number of CVEs reported in a container scan somewhat dramatically (~50 critical, 120 high in a recent scan). I put those in as separate
RUNcommands for clarity but there is a potential argument to combining them into the previous layer if you're trying to optimize for layer count.