I've seen this before on ebony and primarily rosewood fretboards. (Some of them mine.) I am guessing, but I think it has to do with a chemical reaction between your body oil and salt/sweat (maybe even dead skin) causing this to build up in the grain marks in the wood. And I think it's only in the grain because it gets wiped away when you're playing on the flat parts of the wood, on the surface level. I highly suspect this is only a cosmetic issue, but as you say, it is annoying to look at.
Since you've already tried oil, how about rubbing in a good fretboard cleaner first, then oil again, taking your time and rubbing in really good. Use a soft old toothbrush, while 0000 steel wool won't damage anything, it's freaking messy, trust me.
At least my suggestion can do no harm and you'll have a clean, well oiled fretboard!