When CVSplit is created like this:

CVSplit(5, random_state=123)

some devs (including me! 😉) might think, that random_state will cause data to be shuffled.
But it isn't.

I suggest to throw error when random_state is not used for given CVSplit (i.e. when cv parameter is not float).

It is consistent with scikit-learn:

from sklearn.model_selection import KFold
a = KFold(5, shuffle=False, random_state=123)

gives a FutureWarning:

.../lib/python3.7/site-packages/sklearn/model_selection/_split.py:296:
FutureWarning: Setting a random_state has no effect since shuffle is False.
This will raise an error in 0.24.
You should leave random_state to its default (None), or set shuffle=True.
  FutureWarning