Serotonin (5-HT, 5-hydroxytryptamine) is a mitogen in vascular smooth muscle and vascular reactivity to 5-HT is significantly enhanced in hypertension and atherosclerosis. We have tested the hypothesis that tyrosine kinases, enzymes important for mitogenesis, may play a role in 5-HT-induced vascular smooth muscle contractility. Helical strips of rat carotid artery and aorta denuded of endothelium were mounted in tissue baths for measurement of contractile force. The tyrosine kinase inhibitor genistein (5 x 10(-6) M) decreased the potency of 5-HT approximately 4-fold and reduced maximal contraction to 5-HT in carotid arterial strips denuded of endothelium (58% control). Genistein's inactive congener daidzein (5 x 10(-6) M) did not reduce maximal contraction to 5-HT in carotid arteries but did shift the 5-HT concentration response curve 3-fold to the right. Tyrphostin 23 (5 x 10(-5) M), another tyrosine kinase inhibitor, decreased the potency of 5-HT 4-fold and reduced the maximal contraction to 5-HT in the carotid artery (10% control). Contractions induced by phorbol-12,13-dibutyrate (10(-9) to 10(-5) M) were not reduced or shifted by either tyrosine kinase inhibitor, indicating that phorbolester-sensitive protein kinase C isoforms were not affected. KCl-induced contraction was shifted 2-fold and the maximum significantly inhibited by tyrphostin 23 (38.6% control) but not genistein or daidzein, indicating that tyrphostin 23 but not genistein may inhibit voltage-gated calcium channels to reduce contractility. Western blot analysis using antiphosphotyrosine antibody confirmed that 5-HT produced a time- and concentration-dependent increase in the phosphotyrosine immunoreactivity of a 42-kD protein in cultured aortic smooth muscle cells. Lysate immunoprecipitation with an antimitogen-activated-protein (MAP)-kinase antibody indicated that the 42-kD protein was most likely a MAP kinase. 5-HT (10(-5) M) stimulated contraction and increased antiphosphotyrosine immunoreactivity in whole aorta mounted in tissue baths. Importantly, aortic contraction to 5-HT was shifted (5-fold rightward) and reduced (69% control) by genistein but not daidzein. These findings demonstrate that (1) tyrosine kinase activation may partially mediate contractility to 5-HT in arterial smooth muscle, (2) tyrphostin 23 is somewhat nonselective and (3) 5-HT stimulates tyrosine kinase as documented by increased tyrosyl phosphorylation of proteins in cultured aortic smooth muscle cells and aortic tissue in active contraction of 5-HT. These findings have significant implications not only in understanding a novel pathway of 5-HT signal transduction but also in vascular diseases in which growth and/or contractility to 5-HT is increased (e.g. hypertension, atherosclerosis).