Nope, adjust away. You should be able to hear it when the mag pull gets to interfering with the 'purity' of your tone.
The pundits have it as you adjust all pickups low, then bring the neck one up to snuff. After that, you adjust any remaining pups to balance the tone when mixed with the neck. I pretty much do that, and so far I've been happy with the results. But as usual, ymmv.
My bro' in law and his wife (dual income, no kids [DINK]) had a feral cat insert itself into their home, taking residence under the (very expensive) couch. It then proceeded to deliver six semi-feral kittens. After a month, with its work being done, it left. The kittens didn't.
They've taken over the entire house. My bro' in law has moved all of his music gear (especially the tube amps) to another house to avoid "killing the cats". They both complain endlessly about their predicament.
While we'all fully realize that there were (and are) several points along the way where this whole mess could have been avoided (or mitigated), the greater family continues to find significant humor in their delightful (ly stupid) situation.