The most traditional answer to your question is " because it never has been considered a sport." That's correct but very unsatisfying.
I also don't buy the argument of head-to-head competiton not being possible. When I think of head-to-head competition, I think of basketball, hockey, tennis, track, boxing, wrestling, and so on. It's one athlete or team against one or more to see who's the fastest or best. By contrast, downhill skiing, speed skating (long track), pole vaulting, javelin, high jump, long jump, discus, hammer throw, weightlifting, etc. lack head-to-head competition and are long established sporting events.
Not all head-to-head competition should be considered sport. I am very opposed to chess or bridge being admitted as Olympic sports -- this was actually considered not long ago. Similarly, video games do not deserve consideration.
There's a line between art and sport. Most Olympic sports and events are clearly on the sport side, such as weightlifting, wrestling, ice hockey, boxing, track, and so on. You have to look very hard to see any artistry in those sports. There are pasttimes that are very artisitic that involve no athletic skill, such as painting, sculpting, stand-up comedy, debate, etc. No athleticism necessary here. Then there's the sports that straddle the art-sport line: gymnastics, synchronized swimming, diving, synchronized diving, freestyle skiing, etc. I put those on the sport side of the line because of the tremendous amount of athleticism required.
Similarly, I consider ballet to be athletic artistry. That puts it on the art side of the art-sport line and therefore not a sport. Having seen my share of ballet, I believe a world-class gymnast could become a passable ballet dancer but a world-class ballet dancer could never become even a decent gymnast. The heavy emphasis on ballet's artistry comes at the expense of the athleticism necessary for serious consideration as a sport. If ballet is a sport, ballroom dancing, performance art, and break dancing must also be considered sport, and they're not.
Incidentally, I'm not picking on ballet. I do like it, but it's no sport. Neither is rhythmic gymnastics or equestrian dressage, both Olympic events, but that's another debate.