Add support for lists in obs

[brianraymor]

Please describe your wishes and possible alternatives to achieve the desired result.

There are two cases in CELLxGENE Discover that permit multiple values to be annotated in obs metadata fields:

• ethnicity
• disease

Currently, these must be modeled as either CSV row encoded strings (to allow a comma delimiter) or strings with a non-comma delimiter such as "||". Both are inelegant.

Would it be possible for AnnData to support a more natural data type such as lists?

[ilan-gold]

Hi @brianraymor I am not sure I follow what you are saying here. Could you maybe provide a small code snippet of what you would like to see? I'm not sure I follow. For example, isn't ethnicity just a column in an obs pandas.DataFrame?

[brianraymor]

When reviewing our use cases with @ivirshup, he shared this fragment for one approach:

import anndata as ad, pandas as pd, pyarrow as pa, numpy as np

obs = pd.DataFrame({
    "disease_ontology_term_id": pa.array([["MONDO:0004604","MONDO:0043004","MONDO:0800349","MONDO:1030008"]]),
    "disease_ontology_term_label": pa.array([["Hodgkin's lymphoma, lymphocytic-histiocytic predominance", "Weil's disease", "atrial fibrillation, familial, 16","mitral valve insufficiency"]]),
})

...

[ilan-gold]

@ivirshup Can you comment? Does this mean the issue is solved? It seems to me that you actually would not want to split the comma here, or not? Maybe you can open a separate issue for arrow extension dtypes @ivirshup as you suggested?

[ivirshup]

From a phone call, it looks like there is some progress here. A generic solution here is support for arrow arrays as dataframe columns. There is an open PR into zarr that would make this much easier on the zarr side (zarr-developers/zarr-python#3613), but hdf5 would still require some doing.

@ilan-gold did you have more thoughts on how to do this in hdf5? We had mentioned use of a void dtype, but I'm realizing that I don't think we can really chunk that. Maybe it can be a vlen array of bytes which anndata deals with decoding. Could also be worth looking at how awkward array handles writing these kinds of types to hdf5.