Even if you use map.remove() and reset the map element contents, a new map created on the same element can't be panned.

http://bl.ocks.org/tmcw/5462489
http://bl.ocks.org/jfirebaugh/5462515

(Only difference between the two is innerHTML = ''.)