We try to understand the geometry of the moduli space of polarized abelian varieties in characteristic p. E.g. the phenomenon that Hecke orbits blow up and down in a rather unpredictable way. Choose a point $x$, corresponding to a polarized abelian variety. We study $C(x)$ consisting of all moduli points of polarized abelian varieties which have the same $p$-adic and $\ell$-adic invariants. This turns out to be a locally closed subset. We discuss properties of these sets, which form a foliation of the related Newton polygon stratum. We give several applications.