Hey there...
IANAE, but I'm having surgery tomorrow and my PS does not ask his patients to wear a vest either. He says that it makes no difference and he's been doing this for many years and has thousands of gynecomastia surgeries under his belt.
I ordered one anyway, just in case I go on a hay ride or something (?) and don't want my stuff bouncing around after surgery. ...but don't think your PS is crazy; not all docs require their patients to wear a vest.