Background Carcinoma of unknown primary site (CUP) has a poor prognosis and the prognostic factors in these patients are not well established. Furthermore, there are no selection criteria for patients who should benefit from chemotherapy. Methods The medical records of 179 CUP patients who were treated at Taipei Veterans General Hospital from 2000 to 2009 were reviewed. Factors associated with survival were determined by Kaplan-Meier analysis. Differences between the groups with and without palliative chemotherapy were analyzed. Results Univariate analysis revealed multiple prognostic factors, including performance status, lung metastasis, number of metastatic organs, serum albumin, corrected serum calcium, lactate dehydrogenase (LDH), sodium, and cholesterol levels, palliative chemotherapy, and white blood cell and lymphocyte counts. Multivariate analysis showed that performance status < 2, serum albumin level ≥ 3.5 g/dl, corrected serum calcium level < 10.7 mg/dl, single metastatic organ, and palliative chemotherapy were independent factors of better prognosis. Patients with better performance status, higher serum albumin, and lower serum LDH levels had significantly greater benefit from palliative chemotherapy. Conclusions Certain patients with unfavorable CUP will have better survival. Identification of patients with unfavorable CUP who could benefit from palliative chemotherapy warrants future prospective studies.