TLDR: When using a ConfigSpace with named hyperparameters, fANOVA raises an exception if the columns of the feature matrix X are not ordered alphabetically by feature name, because cs.get_hyperparams() returns the parameters in alphabetical order, not the order the parameters have been declared.

Here is an example, where X is a pandas DataFrame with one column for each hyperparameter:

hyperparams = [CSH.CategoricalHyperparameter(param_name, choices=np.unique(X[param_name])) for param_name in X.columns]
cs.add_hyperparameters(hyperparams)

This displays the hyperparameters in the right order (i.e., the order of columns in X). Now, following line :

f = fANOVA(X.values, Y.values, config_space=cs)

raises RuntimeError: There are some categoricals missing in the ConfigSpace specification for hyperparameter noise:.

This is due to a call to cs.get_hyperparams() in the fanova.py code, that sorts the hyperparameters alphabetically, which is different from the column ordering in matrix X.

Maybe I missed something, but it should be fixed or mentioned somewhere. More generally, it would be great to directly accept pandas DataFrames instead of numpy arrays, to automatically derive the hyperparameters from the column names.