Always have a test mini or some other object on hand to spray your primer/sealant prior to your actual minis.
However, failing that there are ways I've found to save the mini from the dry/dust/sandpaper effect.
With dry primer use dry, sturdy paint brush and rapidly 'dust-off' the mini.
For sealants that can sometimes make your mini look dusty you'll need to go find another working spray sealant and respray the mini. The application of a new wet coat will take away the dusty look.
I use Krylon spray primer and Krylon matte acrylic coating\sealer. Sometimes it works like a charm, sometimes not. I always shake the can for about 5 minutes straight before I spray. Also, I always spray from about 5 inches away.
I use the Kilz oil-based primer. That stuff sticks to anything. Thin it out with a little paint thinner and then brush it on. Seems to work pretty well.
If I use spray stuff, always use a test mini first.
Not wanting to deal with that (it just takes one bad batch). I've switched to brush on primers and actually have been using both White and Black Gesso, generally used to prep canvas paintings, to good effect.
First post from long time lurker. Can you describe your priming process? I usually spray outside to avoid the mess but if I move the minis inside and the humididty is different then the frosting is likely to happen. If you just ruined a painted mini you could try very gently brushing a little solvent. I don't know what kind of paint is Krylon but some general purpose dichloromethane based paint thinner chould do the trick.
I meant is it acrylic, enamel or something else? Do you take them back in when you're done spraying or you leave them to dry? To be honest I never had this with primer. Frosting is a common problem with varnish though. Generally rapid change of temp/humidity is a bad idea. So if you take them outside and it's colder/warmer than inside leave them for a couple of minutes. After spraying it's better to leave them for much longer - an hour or two. Extreme humidity or temp is most likely going to ruin your work no matter what you do.
I nearly always paint it so the highlights are a bit exaggerated, then seal it and glue on static grass. I don't think it matters if you are not going to paint it then there's nothing to rub off.