Address many unexpected warnings in pytest #1403

andrewlee94 · 2024-05-03T20:21:31Z

Partially Addresses #1363

Summary/Motivation:

This fixes a lot of unexpected warnings being emitted during pytest runs.

Changes proposed in this PR:

Fix instances of non-test classes with names TestX
Fix warnings about regex escapes
Remove a number of redundant tests for surrogate plotting. Also get plotting methods to return the plot objects so we can close them.
Remove some return statements in test methods/functions.

Legal Acknowledgement

By contributing to this software project, I agree to the following terms and conditions for my contribution:

I agree my contributions are submitted under the license terms described in the LICENSE.txt file at the top level of this directory.
I represent I am authorized to make the contributions and grant the license. If my employer has rights to intellectual property that includes these contributions, I represent that I have received permission to make contributions and grant the required license on behalf of that employer.

bpaul4

Everything looks good and the changes/deletions make sense.

lbianchi-lbl · 2024-05-06T19:10:17Z

For reference, this should be a list of the remaining warnings from Pysmo:

idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test__init__08[array]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test__init__08[DataFrame]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test__init__09[array]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_predict_output_01[array]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_predict_output[array]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/kriging.py:354: RuntimeWarning: invalid value encountered in log
    log_like = (0.5 * ns * np.log(ssd)) + (0.5 * lndetcov)

idaes/core/surrogate/pysmo/tests/test_kriging.py: 314 warnings
idaes/core/surrogate/tests/test_pysmo_surrogate.py: 35 warnings
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/kriging.py:540: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    y_prediction[i, 0] = mean + np.matmul(

idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_error_calculation[array]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_error_calculation[array]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_error_calculation[array]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_error_calculation[DataFrame]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_error_calculation[DataFrame]
idaes/core/surrogate/pysmo/tests/test_kriging.py::TestKrigingModel::test_error_calculation[DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_kriging.py:346: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    y_prediction_exp[i, 0] = mean + np.matmul(

idaes/core/surrogate/pysmo/tests/test_kriging.py: 52 warnings
idaes/core/surrogate/tests/test_pysmo_surrogate.py: 1044 warnings
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/kriging.py:592: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    y_pred[i, 0] = self.optimal_mean + np.matmul(

idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test__init__21[array-array]
idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test__init__21[array-DataFrame]
idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test__init__21[DataFrame-array]
idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test__init__21[DataFrame-DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/polynomial_regression.py:359: UserWarning: The maximum allowed polynomial order is 10. Value has been adjusted to 10.
    warnings.warn(

idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test_results_generation_01[array-DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py:1648: FutureWarning: The behavior of array concatenation with empty entries is deprecated. In a future version, this will no longer exclude empty items when determining the result dtype. To retain the old behavior, exclude the empty entries before the concat operation.
    expected_df = pd.concat(

idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test_results_generation_01[array-DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/polynomial_regression.py:1003: FutureWarning: The behavior of array concatenation with empty entries is deprecated. In a future version, this will no longer exclude empty items when determining the result dtype. To retain the old behavior, exclude the empty entries before the concat operation.
    results_df = pd.concat([results_df, pd.Series({"k": beta[0, 0]})], axis=0)

idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test_results_generation_02[array-DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py:1690: FutureWarning: The behavior of array concatenation with empty entries is deprecated. In a future version, this will no longer exclude empty items when determining the result dtype. To retain the old behavior, exclude the empty entries before the concat operation.
    expected_df = pd.concat(

idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test_results_generation_03[array-DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py:1728: FutureWarning: The behavior of array concatenation with empty entries is deprecated. In a future version, this will no longer exclude empty items when determining the result dtype. To retain the old behavior, exclude the empty entries before the concat operation.
    expected_df = pd.concat(

idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test_confint_regression_01
idaes/core/surrogate/pysmo/tests/test_polynomial_regression.py::TestPolynomialRegression::test_confint_regression_02
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/polynomial_regression.py:1452: UserWarning: Polynomial regression generates poor fit for the dataset
    warnings.warn(

idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_basis_generation[array]
idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_basis_generation[DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py:531: RuntimeWarning: divide by zero encountered in log
    expected_output_3 = np.nan_to_num(distance_array**2 * np.log(distance_array))

idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_basis_generation[array]
idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_basis_generation[DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py:531: RuntimeWarning: invalid value encountered in multiply
    expected_output_3 = np.nan_to_num(distance_array**2 * np.log(distance_array))

idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py: 12 warnings
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/radial_basis_function.py:1108: UserWarning: The parameter matrix A in A.x=B is ill-conditioned (condition number > 1e10). The solution returned may be inaccurate or unstable - inspect rmse error. Regularization (if not already done) may improve solution
    warnings.warn(

idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_rbf_predict_output_06[array]
idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_rbf_predict_output_06[DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py:2093: RuntimeWarning: divide by zero encountered in log
    np.nan_to_num(distance_vec**2 * np.log(distance_vec)), results.weights

idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_rbf_predict_output_06[array]
idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py::TestRadialBasisFunction::test_rbf_predict_output_06[DataFrame]
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/tests/test_radial_basis_function.py:2093: RuntimeWarning: invalid value encountered in multiply
    np.nan_to_num(distance_vec**2 * np.log(distance_vec)), results.weights

idaes/core/surrogate/pysmo/tests/test_sampling.py: 10 warnings
idaes/core/surrogate/pysmo/tests/test_sampling_modified.py: 4 warnings
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:183: UserWarning: The returned number of samples is less than the requested number due to repetitions during nearest neighbour selection.
    warnings.warn(

idaes/core/surrogate/pysmo/tests/test_sampling.py::TestCVTSampling::test__init__selection_11[array]
idaes/core/surrogate/pysmo/tests/test_sampling.py::TestCVTSampling::test__init__selection_11[DataFrame]
idaes/core/surrogate/pysmo/tests/test_sampling.py::TestCVTSampling::test__init__creation_11[list]
idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::CVTSamplingTestCases::test__init__creation_11
idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::CVTSamplingTestCases::test__init__selection_13
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:1420: UserWarning: Tolerance too tight. CVT algorithm may take long time to converge.
    warnings.warn(

idaes/core/surrogate/pysmo/tests/test_sampling.py::TestCustomSampling::test_generate_from_dist_normal_enforced_gaussian_bounds
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:1787: UserWarning: Points adjusted to remain within specified Gaussian bounds. This may affect the underlying distribution.
    warnings.warn(

idaes/core/surrogate/pysmo/tests/test_sampling.py: 22 warnings
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:1799: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    var_values[k] = rep_value

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_08
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:391: UserWarning: The following columns were dropped: ['x2', 'y2']
    warnings.warn(warn_str)

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_09
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:391: UserWarning: The following columns were dropped: ['x2']
    warnings.warn(warn_str)

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_10
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:391: UserWarning: The following columns were dropped: ['x1', 'y2']
    warnings.warn(warn_str)

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_11
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:391: UserWarning: The following columns were dropped: ['y1']
    warnings.warn(warn_str)

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_21
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:451: UserWarning: The following columns were dropped: [2]
    warnings.warn(warn_str)

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_22
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:451: UserWarning: The following columns were dropped: [1, 3]
    warnings.warn(warn_str)

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_23
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:451: UserWarning: The following columns were dropped: [1]
    warnings.warn(warn_str)

idaes/core/surrogate/pysmo/tests/test_sampling_modified.py::SamplingMethodsTestCases::test_selection_columns_preprocessing_24
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo/sampling.py:451: UserWarning: The following columns were dropped: [0, 3]
    warnings.warn(warn_str)

idaes/core/surrogate/tests/test_pysmo_surrogate.py: 7824 warnings
  /home/runner/work/idaes-pse/idaes-pse/idaes/core/surrogate/pysmo_surrogate.py:516: DeprecationWarning: Conversion of an array with ndim > 0 to a scalar is deprecated, and will error in future. Ensure you extract a single element from your array before performing this operation. (Deprecated NumPy 1.25.)
    outputs[i, j] = result.model.predict_output(row_data)

codecov-commenter · 2024-05-06T19:42:41Z

Codecov Report

Attention: Patch coverage is 81.81818% with 4 lines in your changes missing coverage. Please review.

Project coverage is 77.88%. Comparing base (bc2f6c0) to head (67a0b4f).
Report is 26 commits behind head on main.

Files	Patch %	Lines
idaes/apps/matopt/opt/pyomo_modeling.py	0.00%	4 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1403      +/-   ##
==========================================
+ Coverage   77.70%   77.88%   +0.18%     
==========================================
  Files         394      394              
  Lines       65049    65061      +12     
  Branches    14383    14383              
==========================================
+ Hits        50546    50675     +129     
+ Misses      11902    11797     -105     
+ Partials     2601     2589      -12

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Andrew Lee added 5 commits May 3, 2024 12:57

Fix grid integration class names

bb51b75

Fixing pytest marks and more TestClass names

71388dd

Fixing some regexes

5a92788

Fixing returns in tests

a98d8b5

Cleaning up surrogte plotting tests

5263895

andrewlee94 requested review from bpaul4, AlexNoring, MAZamarripa, jsiirola, adowling2, rundxdi, dallan-keylogic, lbianchi-lbl and ksbeattie as code owners May 3, 2024 20:21

andrewlee94 self-assigned this May 3, 2024

andrewlee94 added Priority:Normal Normal Priority Issue or PR testing Issues dealing with testing of code labels May 3, 2024

Fixing plotting modifications

f72ae44

blnicho approved these changes May 3, 2024

View reviewed changes

bpaul4 approved these changes May 6, 2024

View reviewed changes

Andrew Lee added 6 commits May 6, 2024 10:56

Fixing some re.escapes

4cb7382

Fixing misused pytest fixtures

1d865a6

Fixing more re.escapes

76fa9d3

Hopefully the last re.escapes

b3c2609

Missed re.escape

b011767

Fixing typo

67a0b4f

lbianchi-lbl merged commit 6fe724a into IDAES:main May 6, 2024
43 checks passed

andrewlee94 deleted the issue_1363 branch May 6, 2024 20:05

dangunter mentioned this pull request May 30, 2024

Remove PySMO warnings in tests #1421

Closed

OOAmusat mentioned this pull request Aug 17, 2024

PySMO: fix warnings #1472

Merged

lbianchi-lbl added the warnings label Aug 23, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Address many unexpected warnings in pytest #1403

Address many unexpected warnings in pytest #1403

andrewlee94 commented May 3, 2024

bpaul4 left a comment

lbianchi-lbl commented May 6, 2024

codecov-commenter commented May 6, 2024 •

edited

Loading

Address many unexpected warnings in pytest #1403

Address many unexpected warnings in pytest #1403

Conversation

andrewlee94 commented May 3, 2024

Partially Addresses #1363

Summary/Motivation:

Changes proposed in this PR:

Legal Acknowledgement

bpaul4 left a comment

Choose a reason for hiding this comment

lbianchi-lbl commented May 6, 2024

codecov-commenter commented May 6, 2024 • edited Loading

Codecov Report

codecov-commenter commented May 6, 2024 •

edited

Loading