Hammer is still a problem and then you have the question about distance between your magnet and your target object. You would probably need to do some experimentation to see what the deal is with how the meat of your hand interacts with the magnetic fields and if your tricks and such could still be pulled off. Remember that magnetic fields are subject to inverse square so ANY added distance is going to affect its ability to work.
So we’re largely talking about the size of the magnet plus the distance and interference involved. That’s a question of experimentation an testing to determine.