Right, but when there's third parties involved which you may not trust (which is almost always going to be the case when talking to users not on your server), e2e's benefit starts becoming a lot more enticing. And while you have a point on out of band key sharing being annoying, it makes sense as a default - especially when content is going across servers. Content should be secure with an opt-out rather than insecure with an opt-in. The latter is just more error prone.

Also: while it's not friction free, apps like signal have shown that you can get verified e2e to be usable for the general population.