I examine the mean consensus time (i.e., exit time) of the voter model in the so-called two-clique graph. The two-clique graph is composed of two cliques interconnected by some links and considered as a toy model of networks with community structure or multilayer networks. I analytically show that, as the number of interclique links per node is varied, the mean consensus time experiences a crossover between a fast consensus regime [i.e., O(N)] and a slow consensus regime [i.e., O(N^2)], where N is the number of nodes. The fast regime is consistent with the result for homogeneous well-mixed graphs such as the complete graph. The slow regime appears only when the entire network has O(1) interclique links. The present results suggest that the effect of community structure on the consensus time of the voter model is fairly limited.