What's the standard way to handle sorting mixed types for Visidata?

The standard way is to convert the column into a known type, and then anything that can't be converted (errors and nulls) become TypedWrappers which are sortable with any type. Does that work acceptably here too?

Yes, that seems like it should work. Should the rank aggregator pick the known type, and if so, which one? Or is it the user who should convert the column?

Since it's not obvious which type to pick, the user can convert the column.

There are two kinds of ranking operations people may want.

1. keycol-based rank within sheet: what is the rank of this row, vs. all rows in the sheet, ranking by the value of its key columns? In this example, the key column is keycol, and the current column col is ignored:

keycol	col	keycol_sheetrank
1	10	1
1	20	1

2	60	2
2	50	2
2	30	2

2. column-based rank within group: when grouping the rows by key columns, what is the rank of this row, within its group? The current column determines the rank. In this example, the current column is col:

keycol	col	col_grouprank
1	10	1
1	20	2

2	60	3
2	50	2
2	30	1

What is a good name for these two aggregators? sheetrank and grouprank? Or maybe rank_key and rank_col?
Any suggestions?

Okay, I implemented a command that adds a column and applies an aggregator to rows after grouping them by key columns. It's addcol-aggregate.

To get this to work with list aggregators, I made a new class ListAggregator for aggregators that return lists. Their most common use would be with addcol-aggregate. Right now the only two ListAggregators are list and rank.

I also tried making a sheetrank aggregator, but it's too different from normal aggregators. Normal aggregators apply to a column, but sheetrank is more for the sheet. So I broke it out into a separate command, addcol-sheetrank.

I'm a bit unsure about the new behavior of the list aggregator when used with addcol-aggregate. Right now, if the input column has cells with Exceptions, they show up in the new column. But the error text shows up on the display, it's not hidden behind an error note. !. So I could use guidance on a couple of issues here:

1. Should these Exceptions be passed through by the list aggregator, or should they be translated to null?
2. If they should be passed through the aggregator, how do I make them look/behave like the original cell with an exception?
   The relevant code is here:
   

   visidata/visidata/aggregators.py

   Line 120 in c6c608e

   vals = [ col.getTypedValue(r) for r in row_group ]

   
   To see it in action, vd sample_data/test.jsonl, then addcol-aggregate list. The key1 and key1_list columns ought to look the same, but the fourth cell in key1_list reads Expecting ':' delimiter: line 1 column 34 (char 33) instead of empty.

There is one detail about the grouping in addcol-aggregate. If the key column holds multiple cells with null, all nulls are grouped together as one group of rows. But if the key column holds multiple error cells, each error cell forms its own unique group of 1 row, even if all the errors have the same traceback text. I didn't design this, it's just how it behaved on sorting. Does that error cell treatment sound reasonable?

Okay, I moved the addcol-sheetrank code into a features/ file.

I had to move the RankAggregator class there too, because it relies on functions that moved with addcol-sheetrank. Because the new file is not just for the addcol-sheetrank functionality, I named it features/rank.py instead of features/addcol_sheetrank.py.

One notable consequence of this move is that the new features/rank.py file modifies vd.aggregators outside of aggregators.py:

Okay, this one is quite old and quite big at this point! I gave it another pass through, and my remaining questions have to do with aggr.listtype (in review comment), and the vd.aggregate_list function, which seems like it might be an unnecessary and confusing API function. I've asked @anjakefala to look over the behavior too. It'll be nice to finally get this one merged; thanks for your patience on this!

Let me look into why I added a custom stdev function with b41afba and then removed it in ec28446. That's probably an oversight by me.

@midichef I think your PR successfully meets your goals. This is such thoughtfully considered work.

Once you have your fnal changes in, I am going to add an update to the guide in this PR, with documentation.

I have one question for my own clarity: what is rank's intended behaviour when there are multiple key columns?

Say

keycol keycol_2 Item
1 10 Pen
1 20 Pencil
2 60 Book
2 50 Pen
2 30 Book