If your fetish does cause distress, then I would recommend seeking a sex therapist to discuss why and how the fetish distresses you. When I say distress, does your desire to wear women’s underwear (and masturbate in them) prohibit you from being aroused and/or climaxing when you aren’t wearing the panties? Are you still turned on by a partner, or by other aspects of sex? Does your fetish interfere with normative functioning, such as are you obsessed to the point of distraction by the thought of wearing another pair of panties? If any of these questions ring true for you, I would say that seeking a professional to discuss what is going on is your best bet. Check out the American Association of Sexuality Educators, Counsellors and Therapists (AASECT) for their list of qualified therapists. You should also consider checking out the Kink-Aware Professional (KAP) Directory. Both are going to provide you with a list of therapists/professionals local to you that specialize in conversations around sex and kink.
Otherwise, I would just like to say that I think that you are a man with one facet of sexual desires out of many. Panty fetishism is just like any other fetish: it is one aspect of sex that turns you on in a very specific way. That, in itself, is no cause for concern.
