I can't reproduce this on stable versions:

julia> @time Tracker.forward((x,y)->x*y, m1, m2);
  0.013572 seconds (2.29 k allocations: 111.667 KiB)

julia> @time Tracker.forward((x,y)->x*y, cpu(m1), cpu(m2));
  0.018829 seconds (2.07 k allocations: 6.967 MiB)

Are you on stable versions or master branches? It might be worth sending a manifest.

The manifest of the environment I tested this in.

(Created by ] add CuArrays Flux in a new environment)

For some reason, the @time macro is kind of misleading here. It displays similar timings as in your test above, but the first call blocks further calculations / the REPL task way longer. That's why I put the timings in comments, rather than using @time in the opening comment.

You could try:

julia> CuArrays.@time CuArrays.@sync Tracker.forward((x,y)->x*y, m1, m2);
  0.011956 seconds (2.29 k CPU allocations: 111.901 KiB) (1 GPU allocation: 36 bytes, 0.65% gc time of which 100.00% spent allocating)

That should get the full computation time (if it doesn't this is a very strange issue). Assuming it does, post the stats here and it'll show if it's a memory issue. Beyond that it might be good to try running under the profiling tools (both Julia and cudanative/nvprof).