If no momentum is imparted to the ball, no oppositely directed momentum will be imparted to the thrower. Going through the motions of throwing has no net effect. If at the beginning of the throw you begin recoiling backward, at the end of the throw when you stop the motion of your arm and hold onto the ball, you stop moving too. Your position may change a little, but you end up at rest. No momentum given to the ball means no recoil momentum gained by you.