Does anyone else feel like their routines just don't line up with other people's?

I'm starting to think I can only really relax around people who need the same amount of quiet time that I do. How do you tell if a mismatch in daily rhythm is a small thing or a sign it won't work?