Living among politically dissimilar others leads individuals to feel left out and ultimately predicts mobility away from an area. However, does living in politically incongruent environment affect how we relate to other people? In two national samples (n = 12,846 and n = 6,316), the congruence between an individual’s ideological orientation and their community’s ideological orientation were examined. Lack of ideological fit with one’s environment was associated with a difficulty to form close relationships and lower perspective taking. Our findings illustrate the psychological effects of living among dissimilar others and possible explanations for how social environments modulate interpersonal relations.