Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combo of duplication action name check and LocalizeAllActions fix #4975

Open
wants to merge 62 commits into
base: main
Choose a base branch
from

Conversation

jafingerhut
Copy link
Contributor

@jafingerhut jafingerhut commented Oct 22, 2024

I have updated this initial comment with the status of the PR after 52 commits. The most recent changes were simply to update the C++ implementation so they worked after #4992 was merged in.

This PR is a combination of the changes in what was originally two separate PRs, but it seems to me easier to manage if they are combined and reviewed as one. The two earlier PRs were #4951 and #4970. Some of the discussion on comments in #4951 may still provide valuable context.

These changes are intended to address the problem where p4c currently allows a user to have identical @name annotations on multiple objects, e.g. on multiple actions. This can cause incorrect P4Info files to be generated, among other likely problems resulting from that root cause. It also addresses a minor bug in the LocalizeAllActions pass that produces wrong @name annotations for some top-level actions. See this slide deck for examples and more details:

The code changes here are based upon this PR, which stalled a while ago: #3228

These changes are noticeably different from that PR, however, in that this one adds a new pass that occurs before the LocalizeAllActions pass, which is the pass where p4c by design can synthesize actions that are duplicates of user-written actions, all of which have identical @name annotations. Unless we change that behavior, any checks for duplicate @name annotations must be done before that pass. Such a change is described verbally in this comment on the PR above: #3228 (comment) but to my knowledge has not been implemented before. This PR is trying to implement that approach.

It passes all of the CI tests, after modifying a handful of P4 programs that had the error being checked for, so they no longer had the error. One of those was preserved to verify that p4c catches the error with the expected error messages.

The new pass is only enabled if one or more of the command line options are given to generate P4Info output files. The duplicate @name annotations checked for in the new pass only cause problems (that I am aware of) if P4Info output files are being generated. This is similar to some existing checks for duplicate names of P4 objects in the control plane serialization code for P4Info.

There have been multiple related issues filed on p4c over several years that this PR might address, at least partially if not completely.

These issues I believe are fully fixed by this PR:

If this pull request is merged, then this earlier (apparently abandoned) PR should be closed:

The issue below should remain open even if this PR is merged, as the issue is really asking questions about what names should be used for controllable P4 objects in a P4Info file. This PR does not change any names in P4Info files -- it only gives errors and prevents P4Info files from being generated when the current names would conflict:

  • Is this the intended effect of @name annotations when generating P4Info files? #2716
    • I have re-read this issue after commit 24 on this PR, and I believe I should not have linked to it from this PR at all. The questions raised in issue 2716 are definitely related to name annotations and how they affect the contents of P4Info files, but if there are any bugs there, this PR is not attempting to change the contents of correctly-generated P4Info files at all -- it is only attempting to give errors when P4Info files should not be generated.

I still need to re-read and think about whether the following issues are fixed by this PR, or whether there are other questions in them that are not fixed by this PR:

jafingerhut and others added 30 commits October 8, 2024 21:54
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
…ce files that are the output of the midend pass of the compiler, because they can include intentional name duplication created in the LocalizeActions pass. + P4-16 source files created by translating a P4-14 source file to P4-16.

Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Explicitly allow a table and action to have the same name, since there
is a test case that was written explicitly to test that this is permitted.

Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
and for some others, update the error messages we expect to see.

Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Andy Fingerhut <andy.fingerhut@gmail.com>
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
…om:jafingerhut/p4c into add-duplicate-hierarchical-name-check-pass
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
so that it prepends a "." to @name annotations of global actions, if
they do not have one already.

Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
…ations

Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
@@ -0,0 +1,185 @@
/*
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chrispsommers @smolkaj This is one of the two new P4 test programs I would recommend you focus your review on. The comments indicate which parts of the program I expect should give errors, and you can look at the corresponding testdata/p4_16_errors_output/actions-with-duplicate-control-plane-names1.p4-stderr file here to see that it does give errors for those: https://github.com/p4lang/p4c/pull/4975/files#diff-30d824dc2073bd520ea90c54fc52935bad924ddc326b1ab76e225e9c94bacae5

@@ -0,0 +1,173 @@
/*
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chrispsommers @smolkaj This is the second of the two new P4 test programs I would recommend you focus your review on. The comments indicate which parts of the program I expect should not give errors, and they do not. You might also want to look through the contents of the expected generated P4Info file named testdata/p4_16_samples_outputs/actions-almost-duplicate-names1.p4.p4info.txtpb here: https://github.com/p4lang/p4c/pull/4975/files#diff-958b36defef5fa0260fb1c4424a06e6f725d05c1853b99070b691a48ee458c7b

…n-action-name-check-and-localizeallactions-fix
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
... the recent annotations infra changes

Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
@kfcripps kfcripps requested review from oleg-ran-amd and removed request for oleg-ran-amd November 14, 2024 18:48
Copy link

@oleg-ran-amd oleg-ran-amd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jafingerhut Thanks for taking cate of this!
I have not completed the tests yet,
Please find a few comments for the C++ part below.

Comment on lines +20 to +21
#include "ir/ir.h"
#include "ir/visitor.h"

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#include <vector>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added in commit 57

visitDagOnce = false;
}
const IR::Node *preorder(IR::P4Parser *parser) override {
stack.push_back(getName(parser));

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we call prune() to prevent the visitor from going deeper into the parser which cannot have actions?

I think, pushing the parser onto the stack can be removed as we never use parser names to build hierarchical action names.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed there should be no need to walk through sub-nodes within a parser.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in commit 57

Comment on lines 57 to 60
const IR::Node *postorder(IR::P4Parser *parser) override {
stack.pop_back();
return parser;
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we apply the change proposed in the previous comment, this overload can be removed, I believe.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in commit 57

Comment on lines 71 to 75
const IR::Node *preorder(IR::P4Table *table) override {
visit(table->annotations);
prune();
return table;
}

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you please elaborate why we need to visit the table's annotations here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It most likely is not needed, and I will try removing it. This looks like something left over from when I started by copying-and-pasting the HierarchicalNames pass, and never noticed it could/should be removed.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left the method, but removed the visit call, but left the call to prune, since there should be no need to traverse the sub-nodes of a P4 table.

class DuplicateActionControlPlaneNameCheck : public Transform {
std::vector<cstring> stack;
/// Used for detection of conflicting control plane names among actions.
string_map<const IR::Node *> actions;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

string_map preserves the insertion order.
This is not needed for this PR as it never iterates through the elements.
Keeping the order also consumes extra resources (std::list + its memory allocations).
What if we replace this with absl::flat_hash_map?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes in commit 57

}

const IR::Node *DuplicateActionControlPlaneNameCheck::postorder(IR::P4Action *action) {
bool topLevel = findContext<IR::P4Control>() == nullptr;

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

bool topLevel = !stack.empty();

has some chances to be a tiny bit more efficient, I believe.
(this requires the previously suggested change for not pushing parsers onto the stack).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed to bool topLevel = stack.empty(); in commit 57

cstring name = absl::StrCat(".", action->name);
checkForDuplicateName(name, action);
} else {
const auto *e0 = nameAnno->expr.at(0);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const auto *e0 = nameAnno->expr.at(0);
const auto *e0 = nameAnno->getExpr(0);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in commit 55, IIRC

const IR::Node *node) {
bool foundDuplicate = false;
auto *otherNode = node;
auto [it, inserted] = actions.insert(std::pair(name, node));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
auto [it, inserted] = actions.insert(std::pair(name, node));
auto [it, inserted] = actions.emplace(name, node);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in commit 57

auto [it, inserted] = actions.insert(std::pair(name, node));
if (!inserted) {
foundDuplicate = true;
otherNode = (*it).second;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
otherNode = (*it).second;
otherNode = it->second;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in commit 57

foundDuplicate = true;
otherNode = (*it).second;
}
if (foundDuplicate) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could probably fold into !inserted condition above

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in commit 57

} else {
cstring name = absl::StrCat(".", action->name);
action->addAnnotationIfNew(IR::Annotation::nameAnnotation, new IR::StringLiteral(name),
false);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

false is default, could be omitted.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed in commit 57

// If the value of the existing name annotation does not
// begin with ".", prepend "." so that the name remains
// global if control plane APIs are generated later.
const auto *e0 = nameAnno->expr.at(0);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
const auto *e0 = nameAnno->expr.at(0);
const auto *e0 = nameAnno->getExpr(0);

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in commit 55, IIRC

cstring name = absl::StrCat(".", action->name);
action->addAnnotationIfNew(IR::Annotation::nameAnnotation, new IR::StringLiteral(name),
false);
auto nameAnno = action->getAnnotation(IR::Annotation::nameAnnotation);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could move declaration into if:

   if (const auto *nameAnno = ...)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed in commit 57

…tion

Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
@jafingerhut
Copy link
Contributor Author

@jafingerhut Thanks for taking cate of this! I have not completed the tests yet, Please find a few comments for the C++ part below.

No worries. This is an issue I noticed way back in 2019, and it has occasionally bugged me over the years whenever I notice it again. Glad if we can get it fixed.

Thanks for the comments. If it is not obvious, C++ is not something I would call myself an expert in. I believe I have addressed the current round of comments completely in commits today.

While I believe these changes address the questions in the initial comment of this issue: #4651

it does not attempt to make any changes regarding Kyle's follow-up comment here: #4651 (comment)

As I have tried to state elsewhere, that is caused by the LocalizeAllActions pass, and it is by design.

I would recommend either:

(a) realize that this is what LocalizeAllActions does, and write later passes, and your target-specific back end code, to take it into account that there can be multiple actions with the same name, but with the changes in this PR, I believe we can guarantee that all actions with the same hierarchical name were created by LocalizeAllActions, not by the P4 developer.

OR

(b) figure out a way in your version of p4c to avoid using the LocallizeAllActions pass. That is not as simple as just commenting out that pass, I don't think.

Copy link
Contributor

@kfcripps kfcripps left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have reviewed only the C++ changes + the testdata/p4_16_errors/actions-with-duplicate-control-plane-names1.p4 test and related outputs. I will let others review the rest of the changes to the tests outputs. Aside from one suggestion, the parts that I have reviewed LGTM.

Comment on lines +91 to +100
/// Indicates whether control plane API generation is enabled.
/// @returns default to false unless a command line option was
/// given explicitly enabling control plane API generation.
virtual bool controlPlaneAPIGenEnabled() const {
if (p4RuntimeFile.isNullOrEmpty() && p4RuntimeFiles.isNullOrEmpty() &&
p4RuntimeEntriesFile.isNullOrEmpty() && p4RuntimeEntriesFiles.isNullOrEmpty()) {
return false;
}
return true;
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@asl @oleg-ran-amd I think that DuplicateActionControlPlaneNameCheck should run for our backend even when P4Info files are not generated. So we should probably override this function to unconditionally return true.

Signed-off-by: Andy Fingerhut <andy_fingerhut@alum.wustl.edu>
…n-action-name-check-and-localizeallactions-fix
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
control-plane Topics related to the control-plane or P4Runtime. core Topics concerning the core segments of the compiler (frontend, midend, parser)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants