Skip to content

Eliminate large .expect files #2233

Open
@rachitnigam

Description

The general philosophy around checking in golden files requires them to be inspected when changes occur. It is not clear to me that files larger than a reasonably small bound are ever realistically checked. This leads to a false sense of security because we technically have "tests" even though we're not really making sure they are correct.

I ran the following command to find all the .expect files that are longer than 300 files:

fd .expect | xargs -I _ wc -l _ | awk '{ if ( $1 > 300 ) print $2,$1 }' | sort -nk2

A couple of recommendations:

  1. If a particular test does not need to be large, let's bring it under the 300 lines threshold
  2. If a particular test does need to be longer, let's figure out why
    1. If it the case that we're just tracking golden outputs (like the output of a computation) and we would never want it to change, let's add a prefix like .freeze.expect to indicate that this output will never change after being committed to the repo.
    2. If the output format is particularly verbose, let's build some tooling to compress and track only the relevant part of the output. For example, if the Queues testing infrastructure can use a tool to generate a shorter representation of the output, let's build that tool.
  3. Let's write a GH bot to enforce this limit on any new .expect files

Backend

tests/backend/verilog/memory-with-external-attribute.expect 628

Frontends

tests/frontend/ntt-pipeline/ntt-4.expect 306
tests/frontend/ntt-pipeline/ntt-4-reduced-2.expect 312
tests/frontend/relay/batch_matmul.expect 321
tests/frontend/relay/max_pool2d.expect 341
tests/frontend/relay/conv2d.expect 423
tests/frontend/relay/softmax.expect 696

YXI and Xilinx

(@nathanielnrn)

tests/xilinx/language-tutorial-iterate.expect 687
tests/xilinx/gen-verilog/language-tutorial-iterate.expect 686
tests/xilinx/gen-verilog/dot-product.expect 1426
tests/xilinx/gen-verilog/vectorized-add.expect 1426
yxi/tests/axi/read-compute-write/seq-vec-add.expect 395
yxi/tests/axi/read-compute-write/seq-mem-vec-add.expect 475
yxi/tests/axi/dynamic/dyn-vec-add.expect 772
yxi/tests/axi/dynamic/dyn-mem-vec-add.expect 848
yxi/tests/axi/read-compute-write/seq-mem-vec-add-axi-wrapped.expect 9119
yxi/tests/axi/dynamic/dyn-mem-vec-add-axi-wrapped.expect 11012

Interpreter

(@EclecticGriffin)

interp/tests/benchmarks/polybench/linear-algebra-symm.expect 318
interp/tests/benchmarks/polybench/linear-algebra-gramschmidt.expect 400
interp/tests/benchmarks/polybench/linear-algebra-2mm.expect 418
interp/tests/benchmarks/polybench/linear-algebra-syr2k.expect 422
interp/tests/benchmarks/polybench/linear-algebra-3mm.expect 576
interp/tests/benchmarks/polybench/linear-algebra-doitgen.expect 752

Queues

(@anshumanmohan)

calyx-py/test/correctness/queues/binheap/fifo.expect 60008
calyx-py/test/correctness/queues/binheap/pifo.expect 60008
calyx-py/test/correctness/queues/fifo.expect 60008
calyx-py/test/correctness/queues/pifo.expect 60008
calyx-py/test/correctness/queues/pifo_tree.expect 60008
calyx-py/test/correctness/queues/sdn.expect 60008
calyx-py/test/correctness/queues/strict_and_rr_queues/rr_queues/rr_queue_2flows.expect 60008
calyx-py/test/correctness/queues/strict_and_rr_queues/rr_queues/rr_queue_3flows.expect 60008
calyx-py/test/correctness/queues/strict_and_rr_queues/rr_queues/rr_queue_4flows.expect 60008
calyx-py/test/correctness/queues/strict_and_rr_queues/rr_queues/rr_queue_5flows.expect 60008
calyx-py/test/correctness/queues/strict_and_rr_queues/rr_queues/rr_queue_6flows.expect 60008
calyx-py/test/correctness/queues/strict_and_rr_queues/rr_queues/rr_queue_7flows.expect 60008
calyx-py/test/correctness/queues/strict_and_rr_queues/strict_queues/strict_2flows.expect 60008
calyx-py/test/correctness/queues/strict_and_rr_queues/strict_queues/strict_3flows.expect 60008
calyx-py/test/correctness/queues/strict_and_rr_queues/strict_queues/strict_4flows.expect 60008
calyx-py/test/correctness/queues/strict_and_rr_queues/strict_queues/strict_5flows.expect 60008
calyx-py/test/correctness/queues/strict_and_rr_queues/strict_queues/strict_6flows.expect 60008
calyx-py/test/correctness/queues/binheap/stable_binheap.expect 80010
calyx-py/test/correctness/queues/nwc_simple.expect 100012
calyx-py/test/correctness/queues/pcq.expect 100012
calyx-py/test/correctness/queues/pieo.expect 100012

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions