During column expressions computation, Perspective maintains a linear, growable memory block for end-to-end C-string storage to minimize heap allocation, in the form of a t_vocab struct. In investigating failing tests around string values in expressions, @sc1f discovered a bug in the expression engine's use of t_vocab's reallocation method, which is called when the memory block is exhausted. In this case, after an expression was in the process of being calculated, any already-allocated t_scalar structs interned on this t_vocab block would contain invalid pointers to the just-deallocated block.

We discussed this issue and decided to implement a paged-vocab approach that simply allocates a new t_vocab on exhaustion rather than reallocating. Interning strings in the first place is to prevent allocating intermediate to calculating a column; so we should minimize these anyway. There is quite a bit of potentially wasted memory here at the expensive of lookup-time in the string block and other issues; however, in the interest of correctness, we chose this implementation due to its simplicity, practicality and lack of worst-case performance corners.

An alternate approach, updating the scalars in-place by incrementing their pointers by the offset of the reallocated base address, was not implemented; its a place for potential investigation in the future. I expect this will perform better for expressions with a small set strings, but its performance will degrade linearly as this set expands; whether that cost is of practical consequence versus the latter strategy would need performance profiles to know for sure.