💧 👈 Add generalized pointwise loss #540
Conversation
src/pykeen/losses.py
Outdated
| choices=margin_activation_resolver.options, | ||
| ), | ||
| ) | ||
| class GeneralMarginRankingLoss(PairwiseLoss): |
There was a problem hiding this comment.
@mberr what's a better name for this
There was a problem hiding this comment.
Sounds ok. I would go for MarginRankingLoss and rename MRL to HardMarginRankingLoss, but I think this will lead to confusion, since papers do use the term MRL to refer to the hard version.
There was a problem hiding this comment.
Yes I also agree we can't mix with pre-existing nomenclature
There was a problem hiding this comment.
I settled on MarginPairwiseLoss since there are a few that inherit from it in different ways. This also makes it more explicit this only works in pairwise loss world
src/pykeen/losses.py
Outdated
| synonyms = {"Pairwise Hinge Loss"} | ||
|
|
||
| hpo_default: ClassVar[Mapping[str, Any]] = dict( | ||
| margin=dict(type=int, low=0, high=3, q=1), |
There was a problem hiding this comment.
Use a constant for this? the range is chosen somewhat random. btw, why use an integer range for a floating point value.
There was a problem hiding this comment.
🤷 that's what we had before. Would be happy to change if you want to make it something else
src/pykeen/losses.py
Outdated
| Pointwise Loss Activation Margin Formulation Implementation | ||
| =================== ========== ====================== ======================================================== ========================================= | ||
| Hinge ReLU $\lambda \neq 0$ $g(s, l) = \max(0, \lambda -\hat{l}*s)$ :class:`pykeen.losses.PointwiseHingeLoss` | ||
| ??? softplus $\lambda \neq 0$ $g(s, l) = \log(1+\exp(\lambda -\hat{l}*s))$ |
src/pykeen/losses.py
Outdated
| class GeneralMarginRankingLoss(PairwiseLoss): | ||
| r"""Generalized margin ranking loss. | ||
|
|
||
| TODO check order -> is it f(k) - f(\bar{k}) or f(\bar{k}) - f(k)? |
There was a problem hiding this comment.
Okay, let's check:
Larger score means higher plausibility.
So the loss should be large, if the score of the negative one is larger than the score of the positive, i.e., L = f(\bar{k}) - f(k).
If there is a margin, then the positive score should be at least lambda larger than the negative score. If this is not the case, there should be a positive loss. Thus, L = f(\bar{k}) - f(k) + lambda.
| # Utils | ||
| 'loss_resolver', | ||
| ] | ||
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
||
| DEFAULT_MARGIN_HPO_STRATEGY = dict(type=float, low=0, high=3) |
There was a problem hiding this comment.
@mberr here's a reusable constant for the HPO strategy used everywhere except NSSAL
Closes #46, closes #47
This PR generalizes the implementation of
pykeen.losses.SoftplusLossso the Pointwise Hinge Loss could be implemented.It also suggests a new pointwise loss function half way between the pointwise hinge loss and pointwise logistic loss where the softplus activation is used with a nonzero margin. This needs a name, or perhaps someone has already done this and it needs a reference.