In some sense, that's a weaker requirement than to also test how plausible the embedding of x+\Delta t*v is. I could imagine that this leads to problems for a stationary simulation.

Were you thinking of a specific simulation? The steady-state bursty model in your paper? The test doesn't consider the magnitude of the low-d or high-d velocity vector, just the direction. We decided to do it this way because of the various scaling issues of the velocity estimate.

This is an effect of the test statistic being discrete and in line with our expectation of being stochastically greater than or equal to the uniform. I can dig up the plot if you’re interested.

Yes, samples from the null across all cells are close to uniformly distributed (leaning towards conservative with more p-values around 1 and fewer around 0).

If you have a suboptimal cell embedding, velotest can actually yield misleading results, as our null distribution is dependent on this embedding and only alternative velocities in the suboptimal embedding are considered. So, one should first validate the cell embedding before applying velotest.

Yes, especially on the COVID dataset, the embeddings don’t seem to work.

Sorry, I misunderstood you here. I would like to test some more “independent” datasets to conclude this, but the performance seems to degrade quite a bit.

If you want to interpret an embedding, first check that the velocities are properly represented. And yes, this is done on the full data.
However: velotest's results are specific to the underlying gene expression embedding, and some embeddings might be more suitable than others.