Absolutely.
Genetically I’m around 50% descended from people of a culture that I simply wasn’t raised in, which is associated with a religion I never practiced (as opposed to the different religion I left)
Based on looks I could easily identify as that culture and strangers wouldn’t question it. They also don’t question my lack of doing so.
Because if they did consider themself Latino, it might sound more like a personal complaint instead of just a general question.