I have a matrix of GO Term ID's and their GO term names. I'd like to order them by Term size, from largest to smallest number of genes.

So far I have been pulling all the genes from org.Mm.eg.db and counting the number of rows, but this is a slow process and difficult to (l)apply to many GO terms.

nrow(AnnotationDbi::select(org.Mm.eg.db, keytype="GOALL", keys=c("GO:0072089"), columns="SYMBOL"))

Is there a simple way to get the GO size of a list of Mm.db GO Terms so I can arrange them by size?

Try something along the line of the following:

library(GO.db)
library("org.Mm.eg.db")
annotations <- mapIds(org.Mm.eg.db, keys(org.Mm.eg.db, "GO"), "ENSEMBL", "GO", multiVals = "list")

This gives list of all GO term IDs with a vector of all Ensembl genes associated with each term. columns(org.Mm.eg.db) will give you a list of available identifiers you could use instead of Ensembl genes.