Long-duration gamma-ray bursts (LGRBs) are the signatures of extraordinarily high-energy events occurring in our universe. Since their discovery, we have determined that these events are produced during the core-collapse deaths of rare young massive stars. The host galaxies of LGRBs are an excellent means of probing the environments and populations that produce their unusual progenitors. In addition, these same young stellar progenitors makes LGRBs and their host galaxies valuable potentially powerful tracers of star formation and metallicity at high redshifts. However, properly utilizing LGRBs as probes of the early universe requires a thorough understanding of their formation and the host environments that they sample. This review looks back at some of the recent work on LGRB host galaxies that has advanced our understanding of these events and their cosmological applications, and considers the many new questions that we are poised to pursue in the coming years.