It really is hard to overstate the psychological harm that can be done if your phone number falls into unwanted hands.
One of the way that that happens is data-leaks from services that hold your number. Just this month, Twitter has allegedly leaked the numbers of 5.4m users.