Exactly this - Signal's design perpetuates the issue. If they'd not used phone numbers as an identifier, they wouldn't have been there to compromise.
It's a vicious circle as an industry we need to break it.
https://twitter.com/jzikusooka/status/1559232642054344704