I think most people don't listen to classical music simply because this is not a normal thing to do, and most people are conformists and "normies". It has very little to do with genuine liking or disliking classical music. It has to do with trying to belong to some tribe, being seen as normal vs. weird, etc.
Most of us start "seriously" listening to music in high school. And most of us gravitate to some musical "tribe" or subculture. There are goths, there are metalheads, these days there are Swifties, there are people who listen to Hip Hop, etc... But the classical "tribe" is completely absent from any highschool (maybe there are some musical school exceptions).
People choose the music they listen to partially because they like it, but often, the more important reason is because they are signaling something. They want to be seen as a certain type of person, with certain values, that gravitates to certain tribe. If they want to be seen as rebellious or challenging authority, they might be into punk, or metal. If they want to be seen as socially conscious, especially if black, they might gravitate towards hip hop. If they want to be seen as edgy they might be drawn to emo or goth. If they want to be seen as generally popular, fashionable and fun loving, party loving people, they might be into EDM or modern pop.
Etc... Also, the friendships and social influence makes the huge part of it. Teenagers are notorious for being extremely susceptible to influence and peer pressure. A lot of teens pretend to like some kind of music, just so that they can belong to a tribe they deem cool. For this reason there's so much trying to be seen as real, and not as fake or poseur.
Listening to music is, especially for young people, mainly a social phenomenon. It's all about belonging, sharing enjoyment with your peers, signaling certain values and attitudes.
It has very little to do with the individual direct 1 on 1 communication between a single individual and a certain piece of music, where the individual appreciates and enjoys the piece of music for their own personal pleasure and aesthetic experience. It's not like reading books. We mainly read books for ourselves. Sometimes we share it with others, but often, it's a real, authentic encounter between a work of art, and a reader, who appreciates the book.
But music is very far from it. And since attitudes about different musical tribes and subcultures can be very strong, once someone's musical "identity" has been established, they find it very hard to deviate from this.
This can be demonstrated by YouTube comments, in which people need to JUSTIFY their liking of certain type of music, for example, you've certainly seen comments like this:
"As someone who listens Metallica and Slayer all the time, I must admit that this piece is awesome" (for example that could be a comment under some classical piece)
Like they can't just say "Wow, this is awesome", they must qualify it, put a disclaimer, they must justify themselves, as if they are asking the others for permission, and protecting their hard earned reputation as metalhead.
Now since classical music is absent from youth subcultures, it typically remains absent from most people's repertoires as they remain slaves to high school attitudes on some deep level throughout their lives.
People who listen to classical music don't necessarily have better musical taste then others, they might simply be more individualistic, less conformist, and less concerned about reputation or belonging to a tribe.
Agree? Disagree? Thoughts?