Histone deacetylase (HDAC)6 is a unique isoenzyme targeting specific substrates including α-tubulin and heat shock protein (HSP)90. HDAC6 is involved in protein trafficking and degradation, cell shape and migration. Deregulation of HDAC6 activity is associated with a variety of diseases including cancer leading to a growing interest for developing HDAC6 inhibitors. Here, we identified two new structurally related 4-hydroxybenzoic acids as selective HDAC6 inhibitors reducing proliferation, colony and spheroid formation as well as viability of prostate cancer cells. Both compounds strongly enhanced α-tubulin acetylation leading to remodeling of microtubular organization. Furthermore, 4-hydroxybenzoic acids decreased HSP90α regulation of the human androgen receptor in prostate cancer cells by increasing HSP90α acetylation levels. Collectively, our data support the potential of 4-hydroxybenzoic acid derivatives as HDAC6-specific inhibitors with anti-cancer properties.