Currently, two competing models are invoked in order to explain the observable properties of Anomalous X-ray Pulsars (AXPs). One model assumes that AXP emission is powered by a strongly magnetized neutron star - i.e., a magnetar. Other groups have postulated that the unusually long spin periods associated with AXPs could, instead, be due to accretion. As there are severe observational constraints on any binary accretion model, fossil disk models have been suggested as a plausible alternative. Here we analyze fossil disk models of AXPs in some detail, and point out some of their inherent inconsistencies. For example, we find that, unless it has an exceptionally high magnetic field strength, a neutron star in a fossil disk cannot be observed as an AXP if the disk opacity is dominated by Kramers' law. However, standard alpha-disk models show that a Kramers opacity must dominate for the case log B > 12, making it unlikely that a fossil disk scenario can successfully produce AXPs. Additionally, we find that in order to sufficiently spin down a neutron star in a fossil disk, an unusually efficient propeller torque must be used. Such torques are inconsistent with observations of other accreting sytems - particularly High Mass X-ray Binaries. Thus, our analysis lends credence to the magnetar model of AXPs.