Skip to content

should we use polars in sourmash to read manifests and picklists? #3495

Open
@ctb

Description

Seems like "maybe"? :) Should result in a large speedup.

The biggest problem for me is that our biggest CSV files that are routinely encountered tend to be in zip files, which per pola-rs/polars#9601, cannot be nicely loaded with polars. Can we use parquet files instead?

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions