firstly. are you overweight? if you are, then as others have mentioned it may just be a case of pseudogynecomastia. one way that i first realized i had gyne, was when i was lying down, and i felt around my nipple. you could clearly feel the glandular tissue. also some people claim that they feel a (what they call) "muscle" right underneath the nipple.. well if you feel that, its a great chance that its not muscle.
rule of thumb:
gland: lumpy, hard,
fat: soft, no lumps, just soft and amorphous
muscle: muscles aren't too (even when not flexed) very different from gland..
one way of telling muscle from gland (a technique i use) is that i feel the muscle on my upper pectorials, and when i touch my lower pecs (where i believe the gland is) you can totally tell the difference.