Does anyone else experience the 'dual' pull as oddly annoying at first?

I've been trying to figure out my type and keep noticing that the people I get along with best long-term are the ones who initially rubbed me the wrong way. Is that actually how duality feels, or am I just overthinking casual chemistry?