I think you need to see the endo, even if you can't get a referral from your PCP. GP's and family doctors typically don't know what to do about early onset gyne. A typical response is the one you got ("Don't worry about it.....it'll eventually go away.").
And even if your GP does know more than most, in all probability, he/she's probably NOT going to give you a prescription for the medicine(s) that are given to address this problem. They will claim to not deal with that type of thing. But if they care about you, they should give you a referral to the endo, because the endo is generally going to be the doctor that will help you with the problem.
I recently went to an endo for the same problem as you (early onset gyne with pain). He ran some blood tests to check my hormones, but before he even knew the results of the blood work he gave me a prescription for Tamoxifen. That medicine has been beneficial in the treatment of early onset gyne. It usually takes the pain away relatively quickly, and if you catch the gyne early enough, it can be successful in reversing/improving the gyne before the tissue becomes fibrotic. Being a bodybuilder, I'm sure you have heard of it. If you wait too long, however, the medicine will not prove beneficial and surgery will be the only alternative (but only after the active gyne stops and stabilizes).
Another thing endo's give for this problem are aromatase inhibitors, which reduce the amount of estrogen in the body. The estrogen is the reason for the gynecomastia. There are several brands (letrozol, arimidex). Arimidex is often the one prescribed for your problem because it now comes in generic form and it is MUCH cheaper than the non-generic. The non-generics are several hundred dollars.
The thing to be aware of with these drugs are they are typically given to women with breast cancer. Like all meds, they have side effects, some of which can be very bad. For instance, Tamoxifen has been shown to cause blood clots (strokes, pulmonary embolism, deep vein thrombosis), and it is also a known carcinogenic (endometrial cancer in women). The aromatase inhibitors are known to cause bone and joint pain. Each individual has to weigh his options and decide if they want to take these medicines. Are you willing to accept the risk for the potential benefit?