Skip to content

v6.0

Compare
Choose a tag to compare
@bennn bennn released this 20 Jul 17:32

Major Changes Edited all benchmarks so that typed and untyped code are very similar.

If you compare any two typed/A.rkt and untyped/A.rkt files, the only differences should be the requires and the type annotations.

Example: gregor

In at least one place, the untyped gregor code had an extra assert. It's gone now.

diff --git a/benchmarks/gregor/untyped/date.rkt b/benchmarks/gregor/untyped/date.rkt
index a3102a9..6ceccb7 100644
--- a/benchmarks/gregor/untyped/date.rkt
+++ b/benchmarks/gregor/untyped/date.rkt
@@ -63,7 +64,6 @@
 (define date->ymd Date-ymd)
 ;(: date->jdn (-> Any Integer))
 (define (date->jdn d)
-  (unless (Date? d) (error "date->jdn type error"))
   (Date-jdn d))

Example: lnm

Typed lnm now uses asserts instead of casts to validate input data. Untyped lnm uses the same casts.

diff --git a/benchmarks/lnm/typed/spreadsheet.rkt b/benchmarks/lnm/typed/spreadsheet.rkt
index dd2dcbc..b869929 100644
--- a/benchmarks/lnm/typed/spreadsheet.rkt
+++ b/benchmarks/lnm/typed/spreadsheet.rkt
@@ -62,7 +62,7 @@
   (void)
   ;; For each row, print the config ID and all the values
   (for ([(row n) (in-indexed vec)])
-    (void (natural->bitstring (cast n Index) #:pad (log2 num-configs)))
+    (void (natural->bitstring (assert n index?) #:pad (log2 num-configs)))
     (for ([v row]) (void "~a~a" sep v))
     (void)))
 
@@ -71,8 +71,18 @@
 (define (rktd->spreadsheet input-filename
                              #:output [output #f]
                              #:format [format 'tab])
-  (define vec (cast (file->value input-filename) (Vectorof (Listof Index))))
+  (define vec
+    (for/vector : (Vectorof (Listof Index)) ((x (in-vector (assert (file->value input-filename) vector?))))
+      (listof-index x)))
   (define suffix (symbol->extension format))
   (define out (or output (path-replace-suffix input-filename suffix)))
   (define sep (symbol->separator format))
   (vector->spreadsheet vec out sep))
+
+(: listof-index (-> Any (Listof Index)))
+(define (listof-index x)
+  (if (and (list? x)
+           (andmap index? x))
+    x
+    (error 'listof-index)))
diff --git a/benchmarks/lnm/untyped/spreadsheet.rkt b/benchmarks/lnm/untyped/spreadsheet.rkt
index 18be330..6466fb0 100644
--- a/benchmarks/lnm/untyped/spreadsheet.rkt
+++ b/benchmarks/lnm/untyped/spreadsheet.rkt
@@ -14,6 +14,7 @@
 ;; ----------------------------------------------------------------------------
 
 (require
+  "../base/untyped.rkt"
   (only-in racket/file file->value)
   (only-in "bitstring.rkt" log2 natural->bitstring)
 )
@@ -55,7 +56,7 @@
   (void)
   ;; For each row, print the config ID and all the values
   (for ([(row n) (in-indexed vec)])
-    (void (natural->bitstring n #:pad (log2 num-configs)))
+    (void (natural->bitstring (assert n index?) #:pad (log2 num-configs)))
     (for ([v row]) (void "~a~a" sep v))
     (void)))
 
@@ -64,8 +65,16 @@
 (define (rktd->spreadsheet input-filename
                              #:output [output #f]
                              #:format [format 'tab])
-  (define vec (file->value input-filename))
+  (define vec
+    (for/vector ((x (in-vector (assert (file->value input-filename) vector?))))
+      (listof-index x)))
   (define suffix (symbol->extension format))
   (define out (or output (path-replace-suffix input-filename suffix)))
   (define sep (symbol->separator format))
   (vector->spreadsheet vec out sep))
+
+(define (listof-index x)
+  (if (and (list? x)
+           (andmap index? x))
+    x
+    (error 'listof-index)))

results (on Racket 7.7 BC release)

For most benchmarks, performance is the same before & after. But:

  • lnm has lower overhead
  • quadT has higher overhead
  • quadU has higher overhead

lnm

lnm typed code is much faster now (down from ~4.5s to 0.7s) because it uses assert instead of cast. The vector casts in spreadsheet.rkt and summary.rkt cost a little --- putting them back adds 1.5s and 0.5s, respectively. But the big savings comes from replacing (cast .... Index) with (assert .... index?) in bitstring.rkt --- reverting adds almost 2.5s.

quadT
quadU

Both the untyped and fully-typed quad configurations run faster now, which likely makes the mixed configs. look worse. One reason for the change is that quad? is a simple function instead of a define-predicate ... but things are harder to tease apart. (There are few changes to the main files, so things must be happening related to the base/ context, and that's hard to swap out & test.)

Full data & plots here:
gtp-benchmarks-v5-vs-v6.tar.gz

Raw gtp-measure output:
manifest-v6.tar.gz