Heapster must be tested to ensure that it meets our v1.0 scalability goals - 100 node clusters (#3876) each running 30 - 50 pods each (#4188). A soak test might also be very helpful.
Some of the interesting signals to track includes,

• Heapster uptime
• Kubelet stats API latency
• API server watch errors
• Monitoring Backend (GCM) write errors, QPS
• Total number of metrics being handled
• Heapster housekeeping latency

Heapster needs to expose some metrics to aid in scalability testing.

cc @vmarmol