Currently, SpatialConvolutionMM is quite fast on CPU, but its memory requirements are too high.

As it parallellizes the computations over batch examples, it requires a huge buffer (dependent on the batch size) for storing the unfolded image.
I'm reasonably ok with it, as we could share this buffer over multiple convolutions to reduce memory usage (even though it's still requires a lot of memory).
But, as accGradParameters reuses the finput which was already computed in forward (see here and here), this sharing can't be used for training. What is worse, reusing finput in accGradParameters forces us to have another huge buffer fgradInput, of the same size as finput.
Thus, I think memory requirements are too high to use the CPU version of SpatialConvolutionMM in any real case scenario.

What do you think of the following:

• Recompute finput in accGradParameters. This reduce the amount of buffer memory by 2 (no need of fgradInput anymore, and also buffers can be shared between modules. Forward timings stays the same, there is a penalty for backward though

There are other possibilities as well, but which hurts performance even more (but reduces the amount of memory required).

What do you think ?