Is your feature request related to a use case or problem? Please describe.
https://github.com/quantumlib/Cirq/blob/main/cirq-core/cirq/experiments/qubit_characterizations.py practically implements single and two qubit clifford groups in order to do sampling. However the implementation is inefficient taking ~3s to sample 1000 cliffords. that's because the clifford groups are implemented directly as matricies and operations are done as matrix multiplication, inversion or mapping to/from index.

Describe the solution you'd like
update SingleQubitCliffordGate

What is the urgency from your perspective for this issue? Is it blocking important work?
P1 - I need this no later than the next release (end of quarter)