Served over 6 yrs. Army as a Cavalry Scout, could've gone SF, but had young kids at the time, so it wasn't for me. If I could do it all over again, I would've gone Air Force SF. Yes, the air force has a SF unit. All of the armed services have great SF units and they all respect one another (the SF community is very professional and close knit). The difference with the air force is they get the most money to spend on toys, and having worked with a few SF units in my day, I can tell you that they had the best. You'll also live more comfortable when you're not training, because the air force has the best dining and living facilities as well as some of the best looking women. It may be harder to get into, but in my opinion it'll be worth it.
If this is really something that you want to do, then I'd also suggest that you go active duty first and then go reserves to finish out your 8 yr. obligation. I joined up during the first Gulf War, but I didn't go due to it being over with so fast. I have my complaints about the military, but I have a lot of great stories to tell my kids and everywhere I go people can tell that I am a vet. I have friends in the three letter agencies, Somalia vets, and Ranger instructors that I'm all proud of. You already seem like you have the heart. Now it's just taking all the advice you received, making a decision, and mentally preparing yourself for the challenge. Whatever you do, remember it's more of a mental thing and good luck to you.