Is my hobby destroying my chances?

Most people when they look at me think I'm an athlete of some sort. A few times people have asked me if I'm a college basketball player. In reality, I'm a guy who loves video games and learning. I'm a big fan of games and trying to be more open about it now, but it seems if I ever mention them to most people, especially girls, they immediately get turned off. Now I've started being true to myself and saying it anyway even if it makes them lose interest. I'm fairly tall and work out often and where athletic, and baggy clothes so I'm guessing that's why they ume I'm a bball player. Like I said though, I'm much different than my outside appearance. I like learning about history, religions, cultures, and other subjects. I like video games of all genres and I'm also a big fan of music, mostly underground rap. some. It is weird that once I mention what I like they are no longer interested. This question is mostly for women, I'm just wondering if my interests would turn you off.