The very short time scale variability of TeV $\gamma$-ray emission from active galaxies suggests that the acceleration process of particles and the production of primary $\gamma$-rays likely occurs relatively close to the accretion disk. We calculate the $\gamma$-ray spectra produced in an Inverse Compton $e^\pm$ pair cascade initiated by primary $\gamma$-rays which are injected close to the surface of the accretion disk. Possible synchrotron energy losses of secondary cascade $e^\pm$ pairs are also taken into account. Since the soft radiation field is anisotropic, the resulting $\gamma$-ray spectra strongly depend on the observation angle. We investigate their basic properties for different parameters describing such a model. The model is applied to the misaligned blazar Cen A recently detected in the TeV $\gamma$-rays. We conclude on the site of the $\gamma$-ray emission region in Cen A based on the comparison of the model with the observations of this source in the GeV-TeV energy range.