I recently read an article about females who have gained weight and as a result increased their breast size. When they lost weight, their breasts remained large.
I was a big kid but not obese when I started budding, and the docs thought it was a combination of being overweight and pubescent gynecomastia. However they kept growing. I lost some weight and they kept growing. I had a thyroidectomy and they grew even more.
Some of us guys are gonna have breasts and are gonna have to wear a bra. Just like some women are gonna be flat chested and never wear a bra a day in their life.
I'm going off stuff from science quite a while ago, so this may not be current state of knowledge, but IIRC, when one becomes overweight, you gain both adipose cells (fat storage cells) AND those cells increase in size. When you lose weight, you lose the fat stored inside the adipose cells, but the number doesn't decrease. Then it becomes a matter entirely of WHERE the body chooses to preferentially store fat. With above-normal estrogen, or hyper-sensitivity to estrogen, fat will be stored in the breasts and hips, and that's the last place that gives up the fat when weight is lost. So if what I remember is correct, it makes perfect sense that gaining weight = increase in breast size; losing weight = no or minimal loss in size.