Not typically. Supplementary Fig. 6 shows example trajectories of female (red) and male (green) trait means over time at different intersex correlations (rfm). In panel c, for instance, with rfm = 0.95, the two means drift in a correlated fashion near the shared optimum (at 0)

Here’s Fig. 1d: for certain parameter choices, just by chance, the mean trait values in females and males can differ by about half to almost a full phenotypic standard deviation.
👉 Drift alone can create substantial dimorphism.