Pituitary neuroendocrine tumors (PitNETs) are the most common intracranial neoplasms. Although generally benign, they can show a clinically aggressive course, with local invasion, recurrences, and resistance to medical treatment. No universally accepted biomarkers of aggressiveness are available yet, and predicting clinical behavior of PitNETs remains a challenge. In rare cases, the presence of germline mutations in specific genes predisposes to PitNET formation, as part of syndromic diseases or familial isolated pituitary adenomas, and associates to more aggressive, invasive, and drug-resistant tumors. The vast majority of cases is represented by sporadic PitNETs. Somatic mutations in the α subunit of the stimulatory G protein gene (gsp) and in the ubiquitin-specific protease 8 (USP8) gene have been recognized as pathogenetic factors in sporadic GH- and ACTH-secreting PitNETs, respectively, without an association with a worse clinical phenotype. Other molecular factors have been found to significantly affect PitNET drug responsiveness and invasive behavior. These molecules are cytoskeleton and/or scaffold proteins whose alterations prevent proper functioning of the somatostatin and dopamine receptors, targets of medical therapy, or promote the ability of tumor cells to invade surrounding tissues. The aim of the present review is to provide an overview of the genetic and molecular alterations that can contribute to determine PitNET clinical behavior. Understanding subcellular mechanisms underlying pituitary tumorigenesis and PitNET clinical phenotype will hopefully lead to identification of new potential therapeutic targets and new markers predicting the behavior and the response to therapeutic treatments of PitNETs.