I find this highly doubtful. Except for your experience, I have seen absolutely no evidence, scientific, anecdotal or otherwise to support this claim. Psilocybin and Psilocin have very few physical effects even in the short run, and none or almost none in the long run. Their danger is primarily psychological -- they have the potential to trigger latent mental illnesses, and bad trips can lead to personal crises (a la kicka's experience), but as far as we know they do almost nothing to your body.
Hormones can get out of whack for all kinds of reasons, and in many cases gyne is caused by no external factors at all. Your experience is probably just a coincidence. Out of curiosity, how old are you? Did you start taking any new prescriptions around when you developed gyne? What about alcohol/marijuana or steroid use?