I always buy my guitars because I like the tone as they come. The only time I've ever switched from the factory saddle or nut or bridge pins was when they were too worn, a pickup was added, or damaged by some ham-handed person doing a set-up (sometimes that was me early on). I don't recall hearing much difference with other materials, but I get that others hear it. Whatever floats your boat.....
I have replaced Tusq with Tusq on several occasions, and I have made bone compensated saddles too. I have only replaced bridge pins for purely cosmetic reasons.