Have you people never heard of a phone book? Phone numbers aren’t sensitive information. If they want to scrape your phone number they can legally and trivially do so through public data sources. Google does plenty of sketchy things around privacy, but this isn’t one of them, it’s just about security.
Getting over it?