I'm a week post-op now, and I too am questioning the necessity of the compression garment. I think for the first week it was certainly a good idea... but now I'm healing so darn well, with such minimal swelling that it seems like I could discontinue. I would think that my warm showers and daily hour of no vest time would cause everything to start swelling up, but things are pretty stable and look totally fine. Of course, maybe things look so great simply because I've been diligent about wearing my vest... I dunno. Maybe once my doctor sees my healing progress at my follow-up next week, he'll give me a little bit of leeway. Hope so... the second stage vest is a huge pain in the ass to get on and off.
For what it's worth, two other reasons I've read on this forum for using a vest is 1) it helps the tissue adapt properly to your new chest contour and 2) it helps the resulting skin shrink up. Maybe one of the docs will chime in on these points.