Skip to content

[bug] Creating multiple boosters from file results in broken scores. #3778

Closed
@AlbertoEAF

Description

Using the master branch, if I create 2 boosters from file using the C API, I get broken scores from all but the first created booster:

BoosterHandle bh1, bh2;  

LGBM_BoosterCreateFromModelfile("./LightGBM_model.txt", &num_iterations, &bh1); // comment this & you get good results
LGBM_BoosterCreateFromModelfile("./LightGBM_model.txt", &num_iterations, &bh2);

// Score is only correct if I predict with bh1, or predict with bh2 but don't initialize bh1 before initializing bh2:
predict(bh2, values             , NUM_FEATURES, num_iterations, &out_len, &ref_scores[0]); // score changes depending on bh2 being the first Booster created or not.


Meaning, only the first booster created works properly.

I'm on Ubuntu 18.04, x86_64.

Thank you

Activity

changed the title Creating multiple boosters from file results in broken scores. [bug] Creating multiple boosters from file results in broken scores. on Jan 18, 2021
shiyu1994

shiyu1994 commented on Feb 19, 2021

@shiyu1994
Collaborator

@AlbertoEAF It seems that the predict function is not from the C API ? I failed to reproduce the problem with LGBM_BoosterPredictForFile. Loading two models with LGBM_BoosterCreateFromModelfile and then predict with LGBM_BoosterPredictForFile, the two boosters produce consistent results. So could you provide more details about predict function ? Or provide a complete reproducible example ? Thanks.

AlbertoEAF

AlbertoEAF commented on Feb 19, 2021

@AlbertoEAF
ContributorAuthor

Yes @shiyu1994 it's from this test code: https://gist.github.com/AlbertoEAF/5972db15a27c294bab65b97e1bc4c315 :)

If you modify that gist like the snippet above this happens.

shiyu1994

shiyu1994 commented on Feb 20, 2021

@shiyu1994
Collaborator

Sorry @AlbertoEAF , I still cannot reproduce the error. At least, for the attached LightGBM_model.txt file and the following code snippet, using the first and the second booster handle produce exactly the same result.

#include <iostream>
#include <stdio.h>
#include <math.h>
#include <vector>
#include <thread>
#include <ctime>
#include <cstring>
#include "LightGBM/c_api.h"

using namespace std;

#define FMT_HEADER_ONLY
#include "LightGBM/../../external_libs/fmt/include/fmt/format.h"

inline void predict(BoosterHandle handle,
            const void* data,            
            int32_t ncol,                                    
            int num_iterations,            
            int64_t* out_len,
            double* out_result) {
  if (0 != LGBM_BoosterPredictForMatSingleRow(
    handle, 
    data, 
    C_API_DTYPE_FLOAT64,
    ncol, 
    1, // is_row_major
    C_API_PREDICT_NORMAL, 
    0, // start_iteration
    num_iterations, 
    "", 
    out_len, 
    out_result)) {
      throw std::exception();
  }
}

int main(int /*argc*/, char** /*argv*/) {
  BoosterHandle boosterHandle1, boosterHandle2;
  int num_iterations;
  LGBM_BoosterCreateFromModelfile("./LightGBM_model.txt", &num_iterations, &boosterHandle1);
  LGBM_BoosterCreateFromModelfile("./LightGBM_model.txt", &num_iterations, &boosterHandle2);
  cout << "Model iterations " << num_iterations<< "\n";

  double values[] = { 
      0.0, 45.44200,-30.74976,31.78587,4.63569,-15.14894,0.23370,-11.97968,-9.59708,6.48111,-8.89073,4.02405,-2.28873,17.90204,1377.12225,
      1762.04257,947.34430,562.28140,524.26811,361.25410,514.53507,247.70126,399.78880,205.36521,211.86870,44.88971,-299.63923,-227.64169,
      7.34210,-85.80057,16.81211,-41.18581,-12.00858,-41.92117,-26.52312,-12.19765,89.61104,-266.07834,-230.31223,-168.29609,38.40371,31.84016,
      28.98071,-98.73447,26.46565,-30.23272,-1.34637,253.36945,61.73424,16.15765,185.24868,75.57244,55.17367,-41.08869,15.86438,16.03692,-142.57968,
      59.74796,-151.46761,-12.70612,-104.99242,23.10514,47.00179,-13.59894,-79.19045,-27.63745,38.94605,-55.40585,54.73527,15.10521,-3.79596,390.47720,
      17.64018,-68.64827,-62.43611,-31.96095,33.90780,-181.37609,139.84078,-129.48841,76.23826,-8.84046,-0.15439,137.44210,77.54739,-4.22875,-61.92657,
      -33.52722,-3.86253,36.42400,7.17309,
      0.0, 45.44200,-30.74976,31.78587,4.63569,-15.14894,0.23370,-11.97968,-9.59708,6.48111,-8.89073,4.02405,-2.28873,17.90204,1377.12225,
      1762.04257,947.34430,562.28140,524.26811,361.25410,514.53507,247.70126,399.78880,205.36521,211.86870,44.88971,-299.63923,-227.64169,
      7.34210,-85.80057,16.81211,-41.18581,-12.00858,-41.92117,-26.52312,-12.19765,89.61104,-266.07834,-230.31223,-168.29609,38.40371,31.84016,
      28.98071,-98.73447,26.46565,-30.23272,-1.34637,253.36945,61.73424,16.15765,185.24868,75.57244,55.17367,-41.08869,15.86438,16.03692,-142.57968,
      59.74796,-151.46761,-12.70612,-104.99242,23.10514,47.00179,-13.59894,-79.19045,-27.63745,38.94605,-55.40585,54.73527,15.10521,-3.79596,390.47720,
      17.64018,-68.64827,-62.43611,-31.96095,33.90780,-181.37609,139.84078,-129.48841,76.23826,-8.84046,-0.15439,137.44210,77.54739,-4.22875,-61.92657,
      -33.52722,-3.86253,36.42400,7.17309
  };
  const size_t NROWS=2;
  const int NUM_FEATURES = 91;
  double ref_scores[NUM_FEATURES * NROWS];

  int64_t dummy_out_len;

  predict(boosterHandle1, values, NUM_FEATURES, num_iterations, &dummy_out_len, &ref_scores[0]);
  fmt::print("Ref scores: {:.6g}\n", ref_scores[0]);
  predict(boosterHandle2, values, NUM_FEATURES, num_iterations, &dummy_out_len, &ref_scores[0]);
  fmt::print("Ref scores: {:.6g}\n", ref_scores[0]);
}

Could you please provide the LightGBM_model.txt file which produces the error? Thanks!
LightGBM_model.txt

shiyu1994

shiyu1994 commented on Feb 24, 2021

@shiyu1994
Collaborator

@AlbertoEAF Could you please recheck this issue and provide a reproducible case? We need to fix this before next release. Thank you! :)

guolinke

guolinke commented on Feb 27, 2021

@guolinke
Collaborator

gentle ping @AlbertoEAF , for this issue blocks the release of 3.2.0.

AlbertoEAF

AlbertoEAF commented on Mar 1, 2021

@AlbertoEAF
ContributorAuthor

Hi there :),

Unfortunately I cannot provide that model file as it is sensitive, neither do I have much time on my hands to try to produce a simpler example this week. I can try it in the weekend though since it's urgent.

AlbertoEAF

AlbertoEAF commented on Mar 6, 2021

@AlbertoEAF
ContributorAuthor

Hello @shiyu1994 and @guolinke I managed to reproduce it. I created a new branch https://github.com/feedzai/LightGBM/tree/test-double-model-load which you can use (has the model and the profile code).

To test:

# Compile
mkdir build && cd build && cmake -DBUILD_PROFILING_TESTS=ON

# Go to main repo folder & run
cd ..
./lightgbm_profile_single_row_predict

you should get the following outputs, when they should be the same for both boosterHandles 1 & 2:

image

and if you comment the creation of boosterHandle1 and its predictions, boosterHandle2 starts spitting the results of boosterHandle1 in the screenshot above.

shiyu1994

shiyu1994 commented on Mar 8, 2021

@shiyu1994
Collaborator

@AlbertoEAF Thanks for your effort. It seems that the file profiling/profile_single_row_predict.cpp is missing from the branch. Can you please upload it ?

AlbertoEAF

AlbertoEAF commented on Mar 8, 2021

@AlbertoEAF
ContributorAuthor

Done! Sorry for that @shiyu1994 :)

shiyu1994

shiyu1994 commented on Mar 9, 2021

@shiyu1994
Collaborator

@AlbertoEAF I think I found the problem. Is your model file produced by an earlier version of LightGBM (earlier than the merge of linear_tree in #3299)? If so, the field is_linear= is missing from the model file. And when loaded by the latest version of LightGBM, the is_linear_ member of tree will be undefined, which causes undefined prediction behavior.

To fix this, let's set the is_linear_ filed to false when it is absent from the model file, for backward compatibility.

I'll open a PR for this.

added a commit that references this issue on Mar 9, 2021
726777e

17 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      [bug] Creating multiple boosters from file results in broken scores. · Issue #3778 · microsoft/LightGBM