Hill diversity profile

[MKanetscheider]

Added hill diversity profile and function to convert Hill numbers into other popular alpha indices (like simpson and shannon). The latter was requested by Francesca and allows to further manipulate a DataFrame to maximize the gained insights. Like for example calculating evenness indices etc...

Closes #...

• CHANGELOG.md updated
• Tests added (For bug fixes or new features)
• Tutorial updated (if necessary)

[grst]

On question I have about Hill numbers if if and how they take into account different cell numbers per patient or other technical confounders. E.g. with shannon entropy, the entropy would simply increase with increasing cell count per group which is why by default we use normalized shannon entropy in scirpy.

I was wondering if something similar is necessary/possible when using the Hill curves.

[MKanetscheider]

On question I have about Hill numbers if if and how they take into account different cell numbers per patient or other technical confounders. E.g. with shannon entropy, the entropy would simply increase with increasing cell count per group which is why by default we use normalized shannon entropy in scirpy.

I was wondering if something similar is necessary/possible when using the Hill curves.

I think I get what you are referring to, but I am not sure if I know the answer to your question. While I was implementing this I was inspired by cdiversity(https://github.com/AI-SysBio/cdiversity/tree/main) and I think they did it quite similar to the implementation you can see here. Furthermore, I think that many paper I read during my thesis did just apply the Hill number formula for multiple diversity orders and plotted the values as a curve (that is essentially what this function here does as well).

However, I had just a look at the vignette from Immcantation and they seem to consider "more" and have a more advanced way to calculate this (https://alakazam.readthedocs.io/en/stable/vignettes/Diversity-Vignette/). Maybe have a look at it and tell me if my implementation here is just straight wrong or maybe it "just" does some oversimplifications?

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[grst]

Your implementation is certainly not wrong. I do think however that it is important to consider the sequencing depth (or the number of cells per sample, in our case). I also think that it's more of an issue with scRNA-seq data than with bulk data because the number of cells can vary quite a lot between samples.

This function from Alakazam seems to address it and it doesn't look too complicated (but obviously more complicated than the current version): https://bitbucket.org/kleinstein/alakazam/src/f7986680439908fd8660dde25074923f34ea93cf/R/Diversity.R#lines-803:951

Would also be curious to hear @FFinotello's opinion on this as she has worked with the diversity indices before more than I have.

[KilianMaire]

Picking up the depth-correction question from this thread

@grst @FFinotello , following up on the LinkedIn exchange. The open question
here since November 2024 has been whether Hill numbers need a depth/coverage
correction the way Shannon does. They do, and I think this is the piece that
has kept the PR from moving.

First, a correction to something I said publicly: tl.alpha_diversity with its
default normalized_shannon_entropy already accounts for depth. I was too
absolute about that. The gap is narrower and specific: the current
hill_diversity_profile here uses the plug-in Hill estimator, which is biased
by sampling depth at every order of q (richness most strongly, but Inverse
Simpson too). Two samples sequenced to different depths will show different
profiles even when the underlying repertoire is identical. That is exactly the
confounder you flagged, and it is more acute for scRNA-seq where cells per
sample vary a lot.

The established fix is the iNEXT framework (Chao et al. 2014; Hsieh, Ma & Chao
2016): estimate Hill numbers and standardize them to a common sample coverage
before comparing, rather than comparing raw plug-in values. This is the same
approach the Alakazam function you linked is based on.

I have implemented these kernels in pure Python in a small package, hillrep
(https://github.com/KilianMaire/hillrep), validated against R iNEXT 3.0.2 to a
relative tolerance of 1e-6. It already reads scirpy AnnData directly (it
collapses .obs into per-group clonotype abundances keyed on clone_id), so
wiring it into a tl function is mostly an adapter over DataHandler.

Proposal. Build on this PR rather than replace its surface:

• keep hill_diversity_profile and convert_hill_table with the same
  signatures, so the conversion to Shannon/Simpson/evenness that Francesca
  asked for stays;
• swap the plug-in engine for the coverage-standardized estimator, so profiles
  are comparable across samples of different depth;
• add a comparability check that warns when two groups cannot be standardized
  to a shared coverage (sometimes the honest answer is that a fair comparison
  is not possible), surfaced rather than silently returning a number;
• add the unit tests the current PR is still missing, using the
  iNEXT-validated values as fixtures.

One decision I would like your call on before I open a PR:

(A) scirpy takes hillrep as an optional dependency (scirpy[hill]), so the
validated kernels live in one place and stay in sync with the R reference; or

(B) the coverage estimator is vendored into _diversity.py, keeping scirpy
dependency-free at the cost of duplicating the math.

hillrep's core is numpy/scipy/pandas only, so (A) is light, but it is your
dependency policy and your call. I am happy to do the work either way and to
follow your docstring and testing conventions (numpydoc,
@DataHandler.inject_param_docs, pre-commit, tests per function).

Happy to hear any concerns on the estimator choice before I start.

[grst]

Closing in favor of #714