The influence of systematic errors on the calculation of the statistical significance of a $\gamma$-ray signal with the frequently invoked Li and Ma method is investigated. A simple criterion is derived to decide whether the Li and Ma method can be applied in the presence of systematic errors. An alternative method is discussed for cases where systematic errors are too large for the application of the original Li and Ma method. This alternative method reduces to the Li and Ma method when systematic errors are negligible. Finally, it is shown that the consideration of systematic errors will be important in many analyses of data from the planned Cherenkov Telescope Array.