During my research into your question, I found out that what you are experiencing is actually a common occurrence for many people. You can do a couple of different things. If you really like a particular sex toy, you can always cover it in a condom when in use to avoid skin on skin contact. Check first to make sure that the condom’s makeup doesn’t won't damage the toy and you should be good to go. If you are now put off from that toy, or concerned that a condom doesn’t minimize most of the risk (which is totally understandable) then there are a couple of other toy options.
You can go the glass route. Glass is nonporous and easy to sanitize. It will limit you in regards to options of toys. Alternately, you can also try the 100% silicone route. Silicone is supposed to be made without some of the components that are in the rubber mix toys. This option can also give you more choices in regards to if you are seeking a dildo, a vibrator, or a shared toy for you and your partner.The third option is hard plastic. It will also lack the components found in rubber. Hard plastic will also have a wider variety of toys available than glass.
I recommend going to a sex toy store like the Pleasure Chest or Hustler Hollywood (or whatever sex shop is similar and available to you in your area). I use those two as examples because I have used them both, and their staff is highly informed and helpful. You can also look online and contact the company regarding the makeup of the toy prior to purchasing. Just be mindful of the recommended products that go with the toy, such as cleaners and lubricants. Good luck!