The issue with the clutch bag is that it doesn't have straps to just put it on your back or shoulder so you can forget about it and not worry. Having to hold it all the time means you have to put it down, which means you might forget it and leave it somewhere with all your stuff in it, or get it stolen, which is easier of course, because it doesn't have straps to hold it to your body, and it's small.
Fashion is nice and all, but not losing your wallet, phone, and keys is SO much more important, imo.