Among the many things that are sometimes cited as potential causes of Gynecomastia are Beta Blockers used to control Hypertension, Diuretics, and drugs used to treat prostate enlargement.
Meaning that late onset Gynecomastia is not at all uncommon.
Whether it is a big deal for you or not is very personal. If it is a priority issue for you, Get the surgery.
Your personal sense of well being is difficult to put a price tag on.
(proofreaders note that I ended the last sentence with a preposition tsk tsk)