Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

semgrep{,-core}: 1.74.0 -> 1.104.0 #355763

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

andrew-myer
Copy link

@andrew-myer andrew-myer commented Nov 13, 2024

Update semgrep to 1.96.0

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@NixOSInfra NixOSInfra added the 12. first-time contribution This PR is the author's first one; please be gentle! label Nov 13, 2024
@ofborg ofborg bot requested review from ambroisie and 06kellyjac November 14, 2024 10:55
# we're in cli
# replace old semgrep with wrapped one
rm ./bin/semgrep
ln -s $out/bin/semgrep ./bin/semgrep
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Has it been removed?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(The postInstall still mentions a $out/bin/semgrep.)

Copy link
Author

@andrew-myer andrew-myer Nov 14, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, the preCheck was failing because there is no ./bin directory or ./bin/semgrep file but $out/bin/semgrep is still being built.

@andrew-myer
Copy link
Author

Hey @ambroisie @06kellyjac I wanted to check if there is anything needed on my end to get this PR approved? Thanks 😄

@ambroisie
Copy link
Contributor

I'd like @06kellyjac to take a look as well before merging.

@06kellyjac
Copy link
Member

nixpkgs-review result

Generated using nixpkgs-review.

Command: nixpkgs-review pr 355763


x86_64-linux

✅ 13 packages built:
  • python311Packages.semgrep
  • python311Packages.semgrep.dist
  • python311Packages.whispers
  • python311Packages.whispers.dist
  • semgrep (python312Packages.semgrep)
  • semgrep.dist (python312Packages.semgrep.dist)
  • whispers (python312Packages.whispers)
  • whispers.dist (python312Packages.whispers.dist)
  • route-detect
  • route-detect.dist
  • semgrep-core
  • swaggerhole
  • swaggerhole.dist

Copy link
Member

@06kellyjac 06kellyjac left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nix-shell:~/.cache/nixpkgs-review/pr-355763]$ semgrep scan --metrics=off --config 'r/generic.unicode.security.bidi.contains-bidirectional-characters'

┌──── ○○○ ────┐
│ Semgrep CLI │
└─────────────┘

Traceback (most recent call last):
  File "/nix/store/2dv3sdczmkg6wjkn5ama5lgzrzhvj22v-python3.11-semgrep-1.96.0/bin/.pysemgrep-wrapped", line 9, in <module>
    sys.exit(main())
             ^^^^^^
  File "/nix/store/4h5y43skn5ra645n818176zdqs17c0b4-python3.12-semgrep-1.96.0/lib/python3.12/site-packages/semgrep/console_scripts/pysemgrep.py", line 52, in main
    sys.exit(semgrep.main.main())
             ^^^^^^^^^^^^^^^^^^^
  File "/nix/store/4h5y43skn5ra645n818176zdqs17c0b4-python3.12-semgrep-1.96.0/lib/python3.12/site-packages/semgrep/main.py", line 115, in main
    from semgrep.cli import cli
  File "/nix/store/4h5y43skn5ra645n818176zdqs17c0b4-python3.12-semgrep-1.96.0/lib/python3.12/site-packages/semgrep/cli.py", line 5, in <module>
    from semgrep.commands.ci import ci
  File "/nix/store/4h5y43skn5ra645n818176zdqs17c0b4-python3.12-semgrep-1.96.0/lib/python3.12/site-packages/semgrep/commands/ci.py", line 21, in <module>
    import semgrep.app.auth as auth
  File "/nix/store/4h5y43skn5ra645n818176zdqs17c0b4-python3.12-semgrep-1.96.0/lib/python3.12/site-packages/semgrep/app/auth.py", line 8, in <module>
    from semgrep.state import get_state
  File "/nix/store/4h5y43skn5ra645n818176zdqs17c0b4-python3.12-semgrep-1.96.0/lib/python3.12/site-packages/semgrep/state.py", line 13, in <module>
    from semgrep.metrics import Metrics
  File "/nix/store/4h5y43skn5ra645n818176zdqs17c0b4-python3.12-semgrep-1.96.0/lib/python3.12/site-packages/semgrep/metrics.py", line 36, in <module>
    from semgrep.rule import Rule
  File "/nix/store/4h5y43skn5ra645n818176zdqs17c0b4-python3.12-semgrep-1.96.0/lib/python3.12/site-packages/semgrep/rule.py", line 17, in <module>
    from semgrep.rule_lang import EmptySpan
  File "/nix/store/4h5y43skn5ra645n818176zdqs17c0b4-python3.12-semgrep-1.96.0/lib/python3.12/site-packages/semgrep/rule_lang.py", line 19, in <module>
    import jsonschema.exceptions
  File "/nix/store/pw8i005sif9wi9v539x7hlyll9f0ykcn-python3.12-jsonschema-4.23.0/lib/python3.12/site-packages/jsonschema/__init__.py", line 13, in <module>
    from jsonschema._format import FormatChecker
  File "/nix/store/pw8i005sif9wi9v539x7hlyll9f0ykcn-python3.12-jsonschema-4.23.0/lib/python3.12/site-packages/jsonschema/_format.py", line 11, in <module>
    from jsonschema.exceptions import FormatError
  File "/nix/store/pw8i005sif9wi9v539x7hlyll9f0ykcn-python3.12-jsonschema-4.23.0/lib/python3.12/site-packages/jsonschema/exceptions.py", line 15, in <module>
    from referencing.exceptions import Unresolvable as _Unresolvable
  File "/nix/store/qh6pxc98061rqx2v6rs057s6c5dbcw4q-python3.12-referencing-0.35.1/lib/python3.12/site-packages/referencing/__init__.py", line 5, in <module>
    from referencing._core import Anchor, Registry, Resource, Specification
  File "/nix/store/qh6pxc98061rqx2v6rs057s6c5dbcw4q-python3.12-referencing-0.35.1/lib/python3.12/site-packages/referencing/_core.py", line 9, in <module>
    from rpds import HashTrieMap, HashTrieSet, List
  File "/nix/store/qbpa6qshrwiljpqszmf4ppv0zbnjfhyk-python3.12-rpds-py-0.18.1/lib/python3.12/site-packages/rpds/__init__.py", line 1, in <module>
    from .rpds import *
ModuleNotFoundError: No module named 'rpds.rpds'

Do we need to add rpds-py?


Edit: ah it's rpds-py which is broken based on the stack trace

cc: @fabaff

@andrew-myer
Copy link
Author

Result of nixpkgs-review pr 355763 run on aarch64-darwin 1

13 packages built:
  • python311Packages.semgrep
  • python311Packages.semgrep.dist
  • python311Packages.whispers
  • python311Packages.whispers.dist
  • semgrep (python312Packages.semgrep)
  • semgrep.dist (python312Packages.semgrep.dist)
  • whispers (python312Packages.whispers)
  • whispers.dist (python312Packages.whispers.dist)
  • route-detect
  • route-detect.dist
  • semgrep-core
  • swaggerhole
  • swaggerhole.dist

@ofborg ofborg bot requested a review from 06kellyjac December 18, 2024 06:11
@slackerbob
Copy link

There is a message in the existing live version, 1.74.0, that it will stop working tomorrow, Jan 10 2025:

1.74.0

⏫  A new version of Semgrep is available. See https://semgrep.dev/docs/upgrading

⏫  Versions prior to 1.76.0 will cease to work with Semgrep.dev after Jan 10th 2025, please upgrade.

@andrew-myer
Copy link
Author

@06kellyjac I looked at this again. Looks like that rpds.rpds error is caused by a mismatch in python versions
/nix/store/2dv3sdczmkg6wjkn5ama5lgzrzhvj22v-python3.11-semgrep-1.96.0/bin/.pysemgrep-wrapped
vs
/nix/store/qbpa6qshrwiljpqszmf4ppv0zbnjfhyk-python3.12-rpds-py-0.18.1/lib/python3.12/site-packages/rpds/init.py

@andrew-myer andrew-myer force-pushed the semgrep_update branch 2 times, most recently from 16adcc5 to 347ee17 Compare January 22, 2025 19:29
@andrew-myer
Copy link
Author

Result of nixpkgs-review pr 355763 run on aarch64-darwin 1

13 packages built:
  • semgrep (python312Packages.semgrep)
  • semgrep.dist (python312Packages.semgrep.dist)
  • whispers (python312Packages.whispers)
  • whispers.dist (python312Packages.whispers.dist)
  • python313Packages.semgrep
  • python313Packages.semgrep.dist
  • python313Packages.whispers
  • python313Packages.whispers.dist
  • route-detect
  • route-detect.dist
  • semgrep-core
  • swaggerhole
  • swaggerhole.dist

@06kellyjac
Copy link
Member

Thanks for pushing this forwards @andrew-myer
I'll try test this a bit later.

What are your thoughts on just getting this 1.96.0 update merged as-is to get something out there vs trying to update to 1.104.0 now in this PR?


Also FYI https://www.opengrep.dev/

@andrew-myer
Copy link
Author

andrew-myer commented Jan 24, 2025

What are your thoughts on just getting this 1.96.0 update merged as-is to get something out there vs trying to update to 1.104.0 now in this PR?

Let me see if 1.104.0 is any more effort than just updating revs and hashes. If it's that easy, I'd say we go with 1.104.0 since 1.96.0 is a few months old now.

Also FYI https://www.opengrep.dev/

Thats awesome, really excited about this project!

@andrew-myer
Copy link
Author

Result of nixpkgs-review pr 355763 run on aarch64-darwin 1

13 packages built:
  • semgrep (python312Packages.semgrep)
  • semgrep.dist (python312Packages.semgrep.dist)
  • whispers (python312Packages.whispers)
  • whispers.dist (python312Packages.whispers.dist)
  • python313Packages.semgrep
  • python313Packages.semgrep.dist
  • python313Packages.whispers
  • python313Packages.whispers.dist
  • route-detect
  • route-detect.dist
  • semgrep-core
  • swaggerhole
  • swaggerhole.dist

@andrew-myer andrew-myer changed the title semgrep{,-core}: 1.74.0 -> 1.96.0 semgrep{,-core}: 1.74.0 -> 1.104.0 Jan 24, 2025
@andrew-myer
Copy link
Author

For some reason the python312Packages.semgrep is still getting the rpds error in nixpkgs-review. I'm able to run normally with

nix run .#python312Packages.semgrep -- scan --metrics=off --config 'r/generic.unicode.security.bidi.contains-bidirectional-characters'

@06kellyjac
Copy link
Member

If you rebase on the latest copy of master it might introduce the issue for your build which is working. If it continues to work I'm not sure how else the issue could be occurring

Also sorry to be a pain but your other commits should either be squashed together or prefixed with semgrep: (and the bump one formatted like other update commits)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants