Make kikuchipy compatible with PyEBSDIndex 0.2 #652

hakonanes · 2023-08-09T15:11:12Z

Description of the change

This PR aims at making kikuchipy compatible with just released PyEBSDIndex v0.2.0 and to exploit its many new features. New features of main interest are:

Improved PC optimization via particle swarm optimization (PSO), allowing broader search limits in the PC values (default is 0.2 or 20% of the detector dimensions; quite wide!)
Hough indexing of all Laue classes (not only FCC or BCC). So far, only FCC and BCC (m-3m), HCP (6/mmm), and the triclinic case (-1) have been tested.

kikuchipy v0.8.* has PyEBSDIndex as an optional dependency with a compatible release specification (~= 0.1.2), so people installing this release from PyPI will only get PyEBSDIndex v0.1.2. However, our conda-forge feedstock only asks for >= 0.1.2... People installing from conda-forge will most likely get PyEBSDIndex v0.2.0, which does not work with the current kikuchipy release. A big problem. So we should get this merged and released as 0.9.0 quickly. I plan to get this done within Tuesday next week.

Progress of the PR

~~What we should do but perhaps don't have time~~:

~~[ ] Add patterns from crystals having symmetries from all Laue classes to the data module~~
~~[ ] Test HI of these new patterns~~
~~[ ] Add examples showing how to index each of these patterns to the docs~~

For reviewers

The PR title is short, concise, and will make sense 1 year later.
New functions are imported in corresponding __init__.py.
New features, API changes, and deprecations are mentioned in the unreleased
section in CHANGELOG.rst.
New contributors are added to release.py, .zenodo.json and
.all-contributorsrc with the table regenerated.

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

…d PSO from PyEBSDIndex 0.2 Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

CSSFrancis · 2023-08-10T12:48:21Z

@hakonanes Have you considered adding https://github.com/apps/review-notebook-app to kikchipy. It is fairly good at showing diffs for jupyter notebooks?

I can review this if you want someone to (superficially) look at the changes.

hakonanes · 2023-08-14T22:51:10Z

Thanks for the suggestions, @CSSFrancis. I'll consider ReviewNB. I'll ask you for a review once the PR's ready!

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

… of phase lists compatibility Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

…e syntax Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

hakonanes · 2023-11-02T09:21:35Z

@CSSFrancis, this work is ready for merging. If you want to have a quick look, that would be very much appreciated! I plan to merge tomorrow Friday.

I'll get to reviewing and answering your stuff in orix, diffsims, and pyxem shortly!

The only outward API changes are the possibility to (listed in the changelog):

pass a reflector list per phase for use in Hough indexing
pass parameters for particle swarm optimization (PSO) of projection/pattern center (detector-sample geometry calibration)
optimizing one projection center at a time with PSO

Other changes:

Tutorial notebooks: use improvements to optimization and indexing accuracy in PyEBSDIndex v0.2; improve explanations of many steps and analyses.
Replace interactive EBSD master pattern 3D plot with a static plot in tutorials for visualization and kinematic simulations. This was needed because PyVista have discontinued support for the Jupyter backend we used, Panel. The currently supported Jupyter backends cannot provide interactive 3D plots. But they're working on it!

hakonanes · 2023-11-02T09:25:13Z

And thank you for suggesting ReviewNB. It sure beats me comparing the changes to rendered tutorial pages in current docs and the newly built docs from a PR by eye!

doc/dev/running_writing_tests.rst

CSSFrancis

@hakonanes The code looks good here! I'll try and go through the Jupyter notebooks as well. I added a couple of suggestions, but none of them are things that would stop this from merging. Seems like a good addition!

CSSFrancis · 2023-11-02T11:03:28Z

kikuchipy/signals/ebsd.py

+            List of phases. The list must correspond to the phase list
+            in the passed.


Suggested change

List of phases. The list must correspond to the phase list

in the passed.

List of phases. The list must correspond to the passed phase list argument.

Our code style says:

Comment lines should preferably be limited to 72 characters.

I consider docstrings to be comment lines. I'll update this sentence in the docs to make this clearer.

CSSFrancis · 2023-11-02T11:09:20Z

kikuchipy/signals/ebsd.py

+        # Check indexer (but not the reflectors)
        _ = _indexer_is_compatible_with_kikuchipy(
            indexer, sig_shape, nav_size, raise_if_not=True
        )
-        if indexer.phaselist != phase_list_pei:
-            raise ValueError(
-                f"`indexer.phaselist` {indexer.phaselist} and the list determined from"
-                f" `phase_list` {phase_list_pei} must be the same"
-            )
+        _ = _phase_lists_are_compatible(phase_list, indexer, raise_if_not=True)


If these functions take any decent time you could consider caching the results but if they are quick I wouldn't worry about it.

An IPython %timeit returned:

_indexer_is_compatible_with_kikuchipy(): 1.61 us +/- 264 ns

_phase_lists_are_compatible(): 54 us +/- 1.51 us

These Hough indexing methods of the EBSD class are not meant to be called hunderds of times in a loop, so I think this is OK.

CSSFrancis · 2023-11-02T11:18:46Z

.github/workflows/tests.yml

@@ -53,6 +50,9 @@ jobs:
    name: ${{ matrix.os }}-py${{ matrix.python-version }}${{ matrix.LABEL }}
    runs-on: ${{ matrix.os }}
    timeout-minutes: 15
+    env:
+      MPLBACKEND: agg
+      PYTEST_ARGS: --pyargs kikuchipy --reruns 2 -n 2 --cov=kikuchipy


Is there a specific test or reason to add in the reruns? It might be better to just add the @pytest.mark.flaky(reruns=2) to the tests which are flaky and might fail?

Yes, the reason was a flaky test on a macOS run (here). The test check availability of datasets on Zenodo. I've experienced some flaky-ness in the past, so I figured this change was overdue.

Co-authored-by: Carter Francis <csfrancis@wisc.edu>

hakonanes · 2023-11-02T12:14:54Z

Now that you've both reviewed and contributed to the docs, is it OK if I add you to the package credits? This includes the kikuchipy/release.py file, the all-contributors table in the README, and in the Zenodo file. You'll be listed as "Carter Francis" with the following entry in the Zenodo file

   {
      "name": "Carter Francis",
      "orcid": "0000-0003-2564-1851",
      "affiliation": "University of Wisconsin Madison"
    },

…2 characters Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

hakonanes · 2023-11-02T16:27:36Z

I plan to release 0.9.0 once this PR is merged and some other smaller things are done 🚀

CSSFrancis · 2023-11-02T16:42:48Z

Now that you've both reviewed and contributed to the docs, is it OK if I add you to the package credits? This includes the kikuchipy/release.py file, the all-contributors table in the README, and in the Zenodo file. You'll be listed as "Carter Francis" with the following entry in the Zenodo file
   {

      "name": "Carter Francis",

      "orcid": "0000-0003-2564-1851",

      "affiliation": "University of Wisconsin Madison"

    },

Yea that works for me :)

hakonanes added 14 commits August 5, 2023 18:33

Silence deprecation warnings from setuptools and traits

d8e47d3

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Remove unnecessary figure decleration in EBSD.hough_indexing()

b97fb24

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Fix test for figures returned by PyEBSDIndex' indexing

d1ae018

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Update PC opt with PSO from PyEBSDIndex 0.2

7e3a0e9

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Describe how to handle multiprocessing/ptest-xdist in dev docs

e63d909

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Require PyEBSDIndex >= 0.2

84de5de

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Fix conversion between EDAX TSL <--> Bruker PC convensions

289baec

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Use improved PC optimization w/ PSO from PyEBSDIndex 0.2 in HI tut.

77b2ee1

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Use PC plane fitting in hybrid indexing tut. made possible by improve…

1bb1610

…d PSO from PyEBSDIndex 0.2 Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Update reference frames tutorial using the fixed EDAX TSL PC conversion

88ebf90

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Make PC moving-screen calibration tutorial compat. w/PyEBSDIndex 0.2

59004af

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Require PyEBSDIndex 0.2.*

e75f951

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Update orientation dep. on PC and PC plane fit tutorials

91e334b

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Start to make EBSD HI methods compatible with PyEBSDIndex 0.2

2b58b83

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

hakonanes added bug Something isn't working enhancement New feature or request documentation This relates to the documentation labels Aug 9, 2023

hakonanes added this to the v0.9.0 milestone Aug 9, 2023

hakonanes marked this pull request as draft August 9, 2023 15:11

hakonanes mentioned this pull request Aug 9, 2023

Support HyperSpy 2.0 #650

Closed

4 tasks

hakonanes added 2 commits October 19, 2023 14:28

Start to make private HI functionality compatible with PyEBSDIndex 0.2

413b283

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Rewrite similarity check for phase lists from orix and PyEBSDIndex

8c9b16a

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

hakonanes force-pushed the support-pyebsdindex-0-2 branch from 0f354b5 to 8c9b16a Compare October 25, 2023 17:24

hakonanes added 5 commits October 25, 2023 19:37

Remove unnecessary test (was testing PyEBSDIndex, not kikuchipy)

989b1b2

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Test passing reflector list in various forms, start on test for check…

d4fc96b

… of phase lists compatibility Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Complete tests of compatible phase lists and reflector lists

fb834b9

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Improve HI tutorial

0c92cf1

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Improve hybrid indexing tutorial

3547c35

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

hakonanes added 5 commits November 1, 2023 17:47

Fix broken PR template links

e3d2bff

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Silence some warnings when building docs locally, improve make commands

2294eee

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Fix links in tutorials

810e8be

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Capitalize DREAM.3D as the creators intended

ec8b585

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Fix incorrect outlier detection in example, otherwise simplify exampl…

e5a30c8

…e syntax Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

hakonanes marked this pull request as ready for review November 1, 2023 16:53

hakonanes added 8 commits November 1, 2023 18:01

Re-introduce virtual frame buffer in docs (because PyVista needs it?)

3e91a04

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Merge branch 'develop' into support-pyebsdindex-0-2

dad8ba0

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Add build: html to RTD sphinx config

a05120f

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Install pandoc with aptitude on RTD server

69b644a

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Use pytest-rerunfailures to rerun flaky tests

542da6c

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Install pandoc version supported by nbconvert on RTD

aefc94e

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Re-introduce virtual frame buffer for PyVista on RTD

95c1344

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Mention possibility to set reflector lists for Hough indexing

f137df1

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

hakonanes mentioned this pull request Nov 2, 2023

Errors when plotting interactively in notebooks via nbsphinx with PyVista/pythreejs #583

Closed

CSSFrancis reviewed Nov 2, 2023

View reviewed changes

doc/dev/running_writing_tests.rst Outdated Show resolved Hide resolved

CSSFrancis approved these changes Nov 2, 2023

View reviewed changes

Update doc/dev/running_writing_tests.rst

7bd0abe

Co-authored-by: Carter Francis <csfrancis@wisc.edu>

hakonanes added 4 commits November 2, 2023 15:25

Make it clear that comment lines and docstrings should be capped at 7…

3697def

…2 characters Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Add @CSSFrancis to package credits

792a35d

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Describe how to maintain package credits in the developer guide

6a8ff6b

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

Remove reference to Twitter in code of conduct

a794fe8

Signed-off-by: Håkon Wiik Ånes <hwaanes@gmail.com>

hakonanes mentioned this pull request Nov 2, 2023

Use RosettaSciIO and adapt to their IO plugin specification architecture #651

Closed

11 tasks

hakonanes merged commit 7e0425a into pyxem:develop Nov 3, 2023
10 checks passed

hakonanes deleted the support-pyebsdindex-0-2 branch November 3, 2023 07:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make kikuchipy compatible with PyEBSDIndex 0.2 #652

Make kikuchipy compatible with PyEBSDIndex 0.2 #652

hakonanes commented Aug 9, 2023 •

edited

Loading

CSSFrancis commented Aug 10, 2023

hakonanes commented Aug 14, 2023

hakonanes commented Nov 2, 2023

hakonanes commented Nov 2, 2023

CSSFrancis left a comment

CSSFrancis Nov 2, 2023

hakonanes Nov 2, 2023

CSSFrancis Nov 2, 2023

hakonanes Nov 2, 2023

CSSFrancis Nov 2, 2023

hakonanes Nov 2, 2023

hakonanes commented Nov 2, 2023 •

edited

Loading

hakonanes commented Nov 2, 2023

CSSFrancis commented Nov 2, 2023

		List of phases. The list must correspond to the phase list
		in the passed.

Make kikuchipy compatible with PyEBSDIndex 0.2 #652

Make kikuchipy compatible with PyEBSDIndex 0.2 #652

Conversation

hakonanes commented Aug 9, 2023 • edited Loading

Description of the change

Progress of the PR

For reviewers

CSSFrancis commented Aug 10, 2023

hakonanes commented Aug 14, 2023

hakonanes commented Nov 2, 2023

hakonanes commented Nov 2, 2023

CSSFrancis left a comment

Choose a reason for hiding this comment

CSSFrancis Nov 2, 2023

Choose a reason for hiding this comment

hakonanes Nov 2, 2023

Choose a reason for hiding this comment

CSSFrancis Nov 2, 2023

Choose a reason for hiding this comment

hakonanes Nov 2, 2023

Choose a reason for hiding this comment

CSSFrancis Nov 2, 2023

Choose a reason for hiding this comment

hakonanes Nov 2, 2023

Choose a reason for hiding this comment

hakonanes commented Nov 2, 2023 • edited Loading

hakonanes commented Nov 2, 2023

CSSFrancis commented Nov 2, 2023

hakonanes commented Aug 9, 2023 •

edited

Loading

hakonanes commented Nov 2, 2023 •

edited

Loading