The breast area is a common place for fatty deposits, even on otherwise lean men.
We were all born with the beginnings of breasts. Little buds of glandular tissue which, In females, will eventually produce milk.
In early puberty, these buds enlarge. Yes, in males too. Actually this happens to about 2/3 of all males.
Somewhere around half the time, this regresses or shrinks back. About 1/3 of all men have at least some breast gland. It may not be noticed, but it can be palpated--- You can feel it. I have heard it described as a lump like a pea, or as a somewhat flattened disc. In either case, it would be directly behind the nipple. Sometimes it has expanded into the surrounding tissue and you can's really tell if it is there or what its extent actually is.
Some people try to distinguish between True and Pseudo Gynecomastia. True Gynecomastia is the presence of glandular growth while Pseudo Gynecomastia is only fatty deposits. This distinction is rather senseless to my thinking. Besides, the usual condition is both glandular and fatty tissue.
The condition of Gynecomastia is not an abnormality, nor is it a disease. It is an embarrassment, Usually nothing more or less than that. In RARE (note the capitals) cases, it can be an important symptom of medical problems. It does not usually have any physical results. The results can sometimes be severe, and they are almost always emotional or psychological.
You need an impartial opinion. I think that the best person to give you such an opinion would be your Dr.
If the issue of breast enlargement is causing you to be shy, or avoid social activities; Then it is bad enough to get surgery. If it is not having those kinds of effects, then surgery is probably not for you.
The only person you really need to please is yourself.
Please note that I frequently use qualifiers such as About, nearly, sometimes, usually, etc. and rather vague fractions. I personally think the condition is under-reported. The numbers are the best approximations I know of but actual hard statistics, that I find believable, are not available.