Codecov Report

Merging #1686 into master will decrease coverage by 0.05%.
The diff coverage is 100.00%.

@@             Coverage Diff             @@
##            master    #1686      +/-   ##
===========================================
- Coverage   100.00%   99.94%   -0.06%     
===========================================
  Files           21       21              
  Lines         3925     3948      +23     
  Branches       788      790       +2     
===========================================
+ Hits          3925     3946      +21     
- Misses           0        1       +1     
- Partials         0        1       +1     

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 29e3877...028444b. Read the comment docs.

@MrMrRobat I reckon your PR also fixes #1734 so #1757 can probably be closed. I let it open in the meantime but maybe you could add my test in your PR and add #1734 in closed issues ;)

@PrettyWood, test I found in #1757 seems to work regardless of changes I made 🤔.
I assume example from #1734 will work as a test, as it reproduce the issue

@MrMrRobat weird 🤔 I just tried with the test on master and it actually raises

>       globalns = sys.modules[cls.__module__].__dict__.copy()
E       KeyError: None

pydantic/main.py:684: KeyError

Ah, sorry, now I get it. I just removed setting model attribute to a module, but kept setting of a module to a model

So, pickling can't be done if model has no global reference somewhere in code. Pickle threats path in __qualname__ of as such reference.

Dynamically created models don't have global references therefore can not be pickled, unless referenced manually.

We can't set global references to each dynamically created model, as it will either override previous created model, or/and lead to memory leaks.

But turns out, it's not trivial task to determine if model was created globally or locally.

It can be done like this in normal conditions:

def is_call_from_module():
    import inspect

    return inspect.stack()[2].function == '<module>'

However, if code is executed in debugger, value of a inspect.stack()[2].function would be evaluate_expression. Obviously, we can't predict any situation like this.

So, we either need a stable way to detect such differences, or think about other implementations.

typing.Generic, for example, doesn't create new classes for concrete types, but instead uses special class instances and therefore can be pickled. However I'm not sure how it can be used for pydantic models.

I finally found the way to determine call from module, but it won't work with compiled code, but it mostly needed in generics, and since they won't work in compiled code anyway, it's not a problem.

However get_caller_module_name doesn't work in compiled code either, and that's bring inconsistency if we will use it in create_model

The question is: do we need that functionality for non-compiled code, or we better stick with the way it will work the same on both versions?

What will work in compiled and non-compiled code:

• Add default value to create_model argument __module__=__name__ ('pydantic.main')
• Add ensure_picklable function wicth will help to ensure that dynamically created models are pickable (all it actually does is create reference in model.__module__ by model.__name__, if it's not used by another object)

import pickle
from pydantic import create_model
# from pydantic.utils import ensure_picklable

# this should not be done this way. Reference should be set globally, not dynamically,
# otherwise there is no guarantee that reference will be created by the time model will be unpickled
# model = ensure_picklable(create_model('FooModel'))
# pickle.dumps(model)  # works

BarModel = create_model('BarModel', __module__=__name__)
pickle.dumps(BarModel)  # works

Additional things that will work in non-compiled code:

• Determine in what module model was created, so this will work without specifying module:

BarModel = create_model('BarModel')
pickle.dumps(BarModel) 

Hi!

Is this blocked? Can we help?

Thank you

@remidebette, I'm trying to figure out consistent way of solving the problem acroas all platforms. Hacky staff with frames seems to not do the job. Hope, I'll have a success with implementing this with __reduce__ or __reduce_ex__ method, but need more time for that.

Until issue is resolved, you can still fix such issues by using create_model only on module level, declaring variable with the name of the model and using __module__ arg like this:

from pydantic import create_model

MyModel = create_model('MyModel', __module__=__name__)

Such models won't have any problems with pickling/forward refs/etc.

For generics you have to manually set reference to base model module, like so:

from typing import Generic, TypeVar

from pydantic.generics import GenericModel

T = TypeVar('T')
class MyGenericModel(Generic[T]. GenericModel):
    t: T

IntModel = MyGenericModel[int]
setattr(MyGenericModel.__module__, IntModel.__name__, IntModel)

Code above might be implemented in your base classe's __class_getitem__ like so:

class MyAutoGenericModel(Generic[T]. GenericModel):
    t: T

    def __class_getitem__(cls, *args, **kwargs):
        created_model = super().__class_getitem__(*args, **kwargs)
        setattr(cls.__module__, created_model.__name__, created_model)
        return created_model

Thanks for the detailed explanation!

this looks great, please could you:

• add a change description
• fix the coverage

@samuelcolvin, master CI is failing again 😕. I guess PR fromatted with different black version was merged and broke CI.

master should now be fixed.

@samuelcolvin, FYI: test_module fixture can be used to test try_assert.py without having a separate command for that:

def test_assert_in_validators(test_module):
    test_module("try_assert.py")

Upd. Actually, in case of try_assert.py, it could be simply renamed to test_assert.py and runned by pytest withoht any problems, just by putting """PYTEST_DONT_REWRITE""" on top of the module

As for current tests, I decided to rewrite them and enhanced create_module and added run_as_module that runs tests in subprocess