You should at least have an "Ignore" or "Decline silently" button. Even telling the person who sent the invite is (IMO) potentially problematic (as it's confirming that person is *actively* using Signal rather than just having installed it previously and forgotten about it)