-
Notifications
You must be signed in to change notification settings - Fork 161
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ZAP Advanced: Add support for additional report types #1320
Conversation
ZAP reports can come in an enhanced "plus" format, which also contains the full request and response headers and payload. This commit adds support for selecting these formats to ZAP Advanced. To enable them, simply add '-r XML-plus' to the parameters in the Scan definition. The XML-plus format is compatible with the regular XML format used by the parser. Signed-off-by: Max Maass <max.maass@iteratec.com>
Signed-off-by: Max Maass <max.maass@iteratec.com>
Signed-off-by: GitHub Actions <securecodebox@iteratec.com>
MegaLinter status:
|
Descriptor | Linter | Files | Fixed | Errors | Elapsed time |
---|---|---|---|---|---|
✅ GIT | git_diff | yes | no | 0.17s | |
eslint | 1 | 1 | 0.64s | ||
✅ JSON | eslint-plugin-jsonc | 2 | 0 | 1.12s | |
✅ JSON | jsonlint | 2 | 0 | 0.56s | |
prettier | 2 | 1 | 0.69s | ||
✅ JSON | v8r | 2 | 0 | 4.88s | |
✅ PYTHON | bandit | 2 | 0 | 0.3s | |
black | 2 | 1 | 0.63s | ||
flake8 | 2 | 57 | 0.41s | ||
isort | 2 | 1 | 0.13s | ||
✅ PYTHON | pylint | 2 | 0 | 1.93s | |
misspell | 6 | 1 | 0.04s | ||
✅ YAML | prettier | 1 | 0 | 0.43s | |
✅ YAML | v8r | 1 | 0 | 1.25s | |
✅ YAML | yamllint | 1 | 0 | 0.11s |
See errors details in artifact MegaLinter reports on CI Job page
Set VALIDATE_ALL_CODEBASE: true
in mega-linter.yml to validate all sources, not only the diff
I found that when using this patch, the ZAP Advanced parser will sometimes run into OOM exceptions. On the cluster I am using, the default resource limit appears to be 200 MB of memory. It seems like there is no way to modify the requested resources for the parser using the values.yaml or the parse definition - is this correct? What would be the accepted / correct way of increasing these limits? (ping @J12934) |
@malexmave parser and hook memory and cpu limits are currently fixed and cannot be changed :( Its possible to add that in the future by adding these attributes to the ParseDefinitions and ScanComlettionHook See also #1305 |
Okay, looks like I'll also have to patch the operator for the current proof of concept deployment as a stopgap measure :(. Thanks for the quick reply. |
Okay, so, one problem with the new report format is that is can get quite large. For example, if you are scanning a single-page application, it may pull the entire JavaScript code into every single request/response pair that it complains about, leading to large result files. For some reason, this seems to be fine for the lurker, but when the parser pulls it down, it will successfully process it, but then fall flat when trying to upload the results: Starting Parser
Fetching result file
Fetched result file
Transformed raw result file into 17 findings
Adding UUIDs and Dates to the findings
Validating Findings. Environment variable CRASH_ON_FAILED_VALIDATION is set to false
The Findings were successfully validated
Updated status successfully
Uploading results to the file storage service
No response received from FileStorage when uploading finding
Error [ERR_FR_MAX_BODY_LENGTH_EXCEEDED]: Request body larger than maxBodyLength limit
at RedirectableRequest.write (/home/app/parser-wrapper/node_modules/follow-redirects/index.js:102:24)
at RedirectableRequest.end (/home/app/parser-wrapper/node_modules/follow-redirects/index.js:127:10)
at dispatchHttpRequest (/home/app/parser-wrapper/node_modules/axios/lib/adapters/http.js:401:11)
at new Promise (<anonymous>)
at httpAdapter (/home/app/parser-wrapper/node_modules/axios/lib/adapters/http.js:48:10)
at dispatchRequest (/home/app/parser-wrapper/node_modules/axios/lib/core/dispatchRequest.js:58:10)
at Axios.request (/home/app/parser-wrapper/node_modules/axios/lib/core/Axios.js:108:15)
at Axios.<computed> [as put] (/home/app/parser-wrapper/node_modules/axios/lib/core/Axios.js:140:17)
at Function.wrap [as put] (/home/app/parser-wrapper/node_modules/axios/lib/helpers/bind.js:9:15)
at uploadResultToFileStorageService (/home/app/parser-wrapper/parser-wrapper.js:28:6) { For reference, the XML file from ZAP is roughly 30 MB in size. Sadly, this failure in the parsing step also prevents the hook step from running, meaning that the files are not sent to DefectDojo :(. If anyone has an idea what is going on here, I'd be open for suggestions. |
Axios seems to have an issue where we are hitting a maximum upload size, even though neither we nor the server set it. This commit explicitly sets the maximum upload size to unlimited, which seems to fix the issue. Signed-off-by: Max Maass <max.maass@iteratec.com>
We discussed the upload issue in todays meeting, and found that it was caused by a problem with Axios. I added a commit that explicitly tells the upload functionality in the parser SDK to accept arbitrarily large uploads, which seems to have fixed the issue for now. |
ZAP Advanced can be parameterized to return different types of reports. This PR adds the "*-plus" formats that have been added to ZAP (including the XML-plus format that currently has an open pull request against the ZAP Extensions repo). The "plus" reports contain all information of the normal report, plus the full request and response headers and payloads.
While the new format has not yet been approved and merged, it can still be used as follows:
extraVolumes
key:extraVolumeMounts
of thezapContainer
section in the same file:-r XML-plus
to the scan parameters of your ZAP-advanced scanNote that this will break all other report types, as it will lead to Kubernetes creating the folder
/home/zap/.ZAP/reports/traditional-xml-plus/
, which will prevent ZAP from unpacking its own set of reports. This issue will go away once the XML-plus report type is officially supported by ZAP and this workaround with the ConfigMap is no longer necessary.The XML-plus report type is compatible with the existing parser. All other report types are (obviously) incompatible, since the parser expects XML.
Checklist