A more nuanced understanding of the complex relationship between ethnic diversity and social cohesion is needed. Ever since Robert Putnam (2007) has put forward the highly contested constrict claim holding that diversity is related to less trust and more social withdrawing, hundreds of follow-up studies across the globe have been conducted. In the present contribution, we investigated the association between diversity and “hunkering down” in the Netherlands, hereby taking into account the role of segregation. Indeed, Uslaner (2012) pointed to local segregation as the true motor of the so-called diversity effects on intergroup relations in general, and trust in others in particular. We did not only investigate objective indicators of diversity and segregation, but also added an “eye of the beholder” perspective by probing into the subjective perceptions of these variables. Specifically, in a stratified community sample of 680 Dutch ethnic-cultural majority members (52% male, mean age 51), we assessed the additive and interactive effects of four variables (objective diversity, perceived diversity, objective segregation, and perceived segregation) at the municipal level in the prediction of three outcomes (generalized trust, ingroup trust, and outgroup trust). The results revealed three interesting patterns. First, neither of the objective indicators of diversity and segregation, nor their interaction effect significantly predicted any type of trust. Second, higher perceptions of diversity and higher perceptions of segregation were negatively associated with outgroup trust (but not with generalized and ingroup trust). Third, and most importantly, there was a significant interaction effect between perceived diversity and perceived segregation, indicating that simultaneous perceptions of high levels of diversity and high levels of segregation were related to the lowest levels of trust in other ethnic-cultural groups. These findings shed a more nuanced light on the diversity debate, showing that perceptions of segregation shape diversity effects. In sum, the present study shows that perceived rather than objective indicators of diversity and segregation matter, and that both diversity and segregation should be taken into account when it comes to social cohesion in general, and trust in particular.