Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add codable interface for python objects, (Ingress|Egress)Ports python bindings, and other elements required for multi-segment. #18

Merged
42 commits merged into from
Jul 12, 2022

Conversation

drobison00
Copy link
Contributor

@drobison00 drobison00 commented Jun 8, 2022

Blocked by #75

  • Adds py::object and PyHolder as codable objects
  • Support for codable python objects adds some additional utility elements, light wrappers around existing python modules, including multi-process shared memory, and pickle.
  • Adds a simple caching mechanism for Python objects on the c++ side which allows for long lived object references, while guaranteeing they're correctly destructed before the python interpreter exits.
  • Adds python binding overloads support for pipeline.make_segment(segment, ["a", "b", "c"], ["x", "y", "z"], init) -- does not directly support ingress / egress only signatures because we can't differentiate on type and using something like argument count has an ambiguous interpretation.
  • Adds IngressPort and EgressPort specializations for py::objects to allow for correct builder.make_edge behavior in Python.
  • Full set of unit tests for all new objects/code paths.

@drobison00 drobison00 requested review from a team as code owners June 8, 2022 21:37
Copy link
Contributor

@ryanolson ryanolson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately, this requested change is to revert segment::Object back to having a pure virtual name() method, which was neither your change nor mine, but I feel like we need to fix the change that ultimately caused protected member variable to pop up.

The other option would be to make m_name private on segment::Object and provided a protected setter. I'm maybe OK with duplicating the std::string storage of the name property for some object that already store a name like Egress/IngressPort, but it still feels silly, since segment::Object::get_node is still a pure virtual method, meaning the classes that derive from segment::Object can provide name storage if needed and provide the override on name().

include/srf/segment/egress_port.hpp Outdated Show resolved Hide resolved
include/srf/segment/ingress_port.hpp Outdated Show resolved Hide resolved
@ryanolson
Copy link
Contributor

Looking into hangs in testing when ingress/egress ports aren't connected.

I would expect you'd see hangs when they are not connected as the "stop" signal, i.e. the closure of the channel is not allowed to propagate.

We could probably do more to detect this condition earlier, i.e. incomplete connections, but we need to work under the constraint that Morpheus allows for unconnected Sources as part of Stage construction, i.e. there are no true Sinks.

@drobison00
Copy link
Contributor Author

Looking into hangs in testing when ingress/egress ports aren't connected.

I would expect you'd see hangs when they are not connected as the "stop" signal, i.e. the closure of the channel is not allowed to propagate.

We could probably do more to detect this condition earlier, i.e. incomplete connections, but we need to work under the constraint that Morpheus allows for unconnected Sources as part of Stage construction, i.e. there are no true Sinks.

Double checking that its not specific to py::object ingress/egress types; but, if its not, this does seem like an oversight. I think it makes sense to emit a warning if something isn't connected, but an infinite hang shouldn't be possible.

@drobison00 drobison00 requested a review from a team as a code owner June 16, 2022 02:11
@drobison00 drobison00 added improvement Improvement to existing functionality non-breaking Non-breaking change improvement labels Jun 16, 2022
@drobison00 drobison00 changed the base branch from main to branch-22.06 June 16, 2022 02:18
@drobison00 drobison00 changed the title Draft: Add codable interface for python objects, (Ingress|Egress)Ports python bindings, and other elements required for multi-segment. Add codable interface for python objects, (Ingress|Egress)Ports python bindings, and other elements required for multi-segment. Jun 16, 2022
Copy link
Contributor

@dagardner-nv dagardner-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, had a few random questions.
Thanks for filling out the pydocs

python/srf/_pysrf/include/pysrf/utilities/object_cache.hpp Outdated Show resolved Hide resolved
python/srf/_pysrf/src/segment.cpp Outdated Show resolved Hide resolved
@ryanolson ryanolson added this to the Multi-Segment Support milestone Jun 29, 2022
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A couple of really small changes

python/tests/test_edges.py Outdated Show resolved Hide resolved
python/tests/test_edges.py Outdated Show resolved Hide resolved
python/tests/test_edges.py Outdated Show resolved Hide resolved
include/srf/node/edge_builder.hpp Outdated Show resolved Hide resolved
python/srf/_pysrf/src/utils.cpp Outdated Show resolved Hide resolved
Copy link
Contributor

@mdemoret-nv mdemoret-nv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Lets make an issue to add the C++ Ingress/Egress optimization but this should be good to test now

@mdemoret-nv
Copy link
Contributor

@gpucibot merge

@ghost ghost merged commit 1a2d701 into nv-morpheus:branch-22.08 Jul 12, 2022
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
improvement Improvement to existing functionality non-breaking Non-breaking change
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants