When your Dr. said you only have fat and nothing to worry about, I believe he meant that you do not have a health issue that is something to be concerned about.
If you have a concern about the appearance of your chest, you should seek the advice of a plastic surgeon with expertise in gynecomastia surgery.
All boys and men have a small mass of fibrous breast tissue immediately below the areolas. This small mass of breast tissue does not become apparent externally unless it grows in size to a point where the nipple-areola complexes begin to bulge outward, hence the term ‘puffy nipples’.
Treatment consists of excision of the excess breast tissue using one or more of a number of techniques: direct excision via an incision along the inferior border of the areola, ‘pull-through’ excision through two or more mini-incisions at the areolar border, and arthroscopy shaver excision via a remote mini-incision (ideally concealed in hair-bearing skin in the underarm area). As there is no fat excess in patients with pure glandular gynecomastia, no liposuction is required.
Care must be taken on the part of the surgeon to perform a precise excision of the excess breast tissue, and to preserve a thin cuff of breast tissue on the posterior surface of the areolas(s). Over-resection will result in areolar retraction and concavity as well as contour irregularity in the area immediately surrounding the nipple-areola complexes.