Skip to content

NumPy 1.14+ compatibility #22

Open
@Stewori

Description

... is currently broken:

Traceback (most recent call last):
  File "/data/workspace/linux/JyNI/JyNI-Demo/src/JyNINumPyTest.py", line 68, in <module>
    import numpy as np
  File "/data/workspace/linux/numpy/1.14.0/numpy/__init__.py", line 168, in <module>
    from . import ma
  File "/data/workspace/linux/numpy/1.14.0/numpy/ma/__init__.py", line 44, in <module>
    from . import core
  File "/data/workspace/linux/numpy/1.14.0/numpy/ma/core.py", line 6333, in <module>
    masked = masked_singleton = MaskedConstant()
  File "/data/workspace/linux/numpy/1.14.0/numpy/ma/core.py", line 6270, in __new__
    cls.__singleton = MaskedArray(data, mask=mask).view(cls)
  File "/data/workspace/linux/numpy/1.14.0/numpy/ma/core.py", line 2790, in __new__
    _data = ndarray.view(_data, cls)
AttributeError: 'type$1' object has no attribute 'update'

Stick to NumPy 13.3 until this is solved...

Activity

changed the title NumPy 1.14.0 compatibility NumPy 1.14 compatibility on Jun 30, 2018
Stewori

Stewori commented on Jun 30, 2018

@Stewori
OwnerAuthor

I just investigated newer NumPy versions. 14.1 and 14.2 result in the same error as 14.0 above.
NumPy 14.3, 14.4, 14.5 give this new error:

('1.8.0_171', 'Oracle Corporation', ('OpenJDK 64-Bit Server VM', '25.171-b11', 'Oracle Corporation'), ('Linux', '4.4.0-128-generic', 'amd64'))
Traceback (most recent call last):
  File "/data/workspace/linux/JyNI/JyNI-Demo/src/JyNINumPyTest.py", line 73, in <module>
    import numpy as np
  File "/data/workspace/linux/numpy/1.14.5/numpy/__init__.py", line 166, in <module>
    from . import random
  File "mtrand.pyx", line 1, in init mtrand
  File "/data/workspace/linux/numpy/1.14.5/numpy/__init__.py", line 118, in <module>
    __NUMPY_SETUP__
TypeError: unhashable type: 'str'
Stewori

Stewori commented on Jul 1, 2018

@Stewori
OwnerAuthor

So, I located the first type of failure to NumPy's source file multiarray/ctors.c.
Specifically in the function PyArray_NewFromDescr_int the line res = PyObject_Call(func, args, NULL); fails, i.e. returns zero, triggering the subsequent goto fail;. Next step is to identify why this call fails. For now I only know that it emits a bunch of Python-exceptions, where the message that we see, i.e. 'type$1' object has no attribute 'update' is only the tip of the iceberg. The full sequence of error messages is:

'numpy.ndarray' object has no attribute '_fill_value'
'numpy.ndarray' object has no attribute '_hardmask'
'numpy.ndarray' object has no attribute '_sharedmask'
'numpy.ndarray' object has no attribute '_isfield'
'numpy.ndarray' object has no attribute '_baseclass'
'type$1' object has no attribute 'update'

So far I conclude that somehow attribute (write-)access of numpy.ndarray got messed up.

Stewori

Stewori commented on Jul 2, 2018

@Stewori
OwnerAuthor

Tracked it down a bit further:
It fails in the method MaskedArray._update_from in ma/core.py.
Specifically the line self.__dict__.update(_dict) fails. Here _dict contains a bunch of updates for attributes that shall be written to self.__dict__. It seems that for some reason self.__dict__ might not be an ordinary dictionary, which somehow causes the issue.
type(self.__dict__) results in <type 'type$1'>, which is familiar from our error message 'type$1' object has no attribute 'update'. print(self.__dict__) outputs <attribute '__dict__' of 'MaskedArray' objects>. So maybe it's an issue with attribute wrappers while the dictionary itself is fine. It seems like the key lies in understanding what type$1 actually is.

The MaskedArray class extends ndarray and I remember that I once observed Astropy to fail with JyNI because of ndarray subclassing. Maybe before version 14.0 the NumPy import worked without subclassing ndarray. Unfortunately the commit history of ma/core.py is insanely complex and it is challenging to track down the breaking commit. I thought about using git's bisect feature, but that's probably tedious because of all the rebuilds required. For now I will just continue investigation. If I can fix ndarray subclassing in JyNI, then it hopefully also improves usecases like in Astropy.

Stewori

Stewori commented on Jul 4, 2018

@Stewori
OwnerAuthor

Note that the CPython output is as one would naively expect:
Instead of <type 'type$1'> we get <type 'dict'> and instead of <attribute '__dict__' of 'MaskedArray' objects> we get {}. So somehow JyNI, or maybe Jython already, messes up this attribute. Maybe it just misses to evaluate the attribute wrapper properly.

Stewori

Stewori commented on Jul 5, 2018

@Stewori
OwnerAuthor

The "dict" object of type <type 'type$1'> is an instance of PyDataDescriptor, see PyDataDescriptor.java. type$1 is a generated type (as I already expected actually). PyDataDescriptor is abstract and Jython generates concrete implementations at runtime, resulting in these strange type names. Now the real question is why JyNI does not evaluate __get__. Maybe the case of extending natively defined builtin-like types is not yet properly covered, an observation that is consistent with the way how Astropy fails.
I will review the way how new-style classes are handled by JyNI. Originally the described case should have been workable, but that appears to be incomplete. Maybe this will require something like PyCPeerDerived. We'll see...

Stewori

Stewori commented on Jul 5, 2018

@Stewori
OwnerAuthor

Quick experiment:
Inserted this code snippet into PyCPeer.__findattr_ex__ right before the return statement:

if (er != null && er.implementsDescrGet() && er.isDataDescr()){
    er = er.__get__(this, objtype);
}

Now the descriptor is evaluated, but so far this doesn't fix anything because it just evaluates to None. It results now in AttributeError: 'NoneType' object has no attribute 'update' instead of AttributeError: 'type$1' object has no attribute 'update' (Is this an improvement?).
It seems like the generated implementation of PyDataDescriptor does not know what to do with a PyCPeer...

changed the title NumPy 1.14 compatibility NumPy 1.14+ compatibility on Jul 26, 2018
Stewori

Stewori commented on Jul 26, 2018

@Stewori
OwnerAuthor

The recently released NumPy 1.15.0 gives the following error on import:

/usr/lib/jvm/java-8-openjdk-amd64/bin/java:
symbol lookup error:
/data/workspace/linux/numpy/1.15.0/numpy/core/multiarray.so:
undefined symbol:
PyStructSequence_InitType

However, I suppose this is easier to fix. Adding a PyStructSequence_InitType implementation is not a big deal. However I suspect there will be subsequent issues.

ShaharZivan

ShaharZivan commented on Nov 27, 2018

@ShaharZivan

Did you find any solution for this yet? Even a temporary hardcoded patch would do. I need to use sequitur-g2p with Jython and sadly it requires NumPy 1.14.2 or higher.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      NumPy 1.14+ compatibility · Issue #22 · Stewori/JyNI