should we use polars in sourmash to read manifests and picklists? #3495
Open
Description
Seems like "maybe"? :) Should result in a large speedup.
The biggest problem for me is that our biggest CSV files that are routinely encountered tend to be in zip files, which per pola-rs/polars#9601, cannot be nicely loaded with polars. Can we use parquet files instead?
Metadata
Assignees
Labels
No labels