Background Pharmacists can play a key role in managing ailments through their primary roles of supplying over-the-counter (non-prescription) medicines and advice-giving. It must be ensured that pharmacy staff practise in an evidence-based, guideline-compliant manner. To achieve this, mystery shopping can be used as an intervention to assess and train pharmacy staff. Objective To determine if repeated student pharmacist mystery shopping with immediate feedback affected the outcome of scenarios requiring referral to a medical practitioner. To determine what, if any, factors may influence whether referral occurred. Setting Thirteen community pharmacies across metropolitan Sydney, Australia. Methods Sixty-one student pharmacist mystery shoppers visited 13 community pharmacies across metropolitan Sydney once weekly over nine weeks between March–October 2015 to conduct audio-recorded mystery shopping visits with assigned scenarios (asthma, dyspepsia, diarrhoea). Students returned to the pharmacy immediately to provide staff members with feedback. Pharmacy staff were scored by mystery shoppers according to a standardised scoresheet. Score data and other characteristics, such as the assigned scenario, were analysed via correlation and logistic regression modelling. Main outcome measure Whether a student mystery shopper was appropriately referred to a medical practitioner based on the presenting symptoms. Results 158 visits were eligible for analysis. Referral to a medical practitioner was appropriately made in 66% of visits. The regression model provided an R2 value of 0.73; the questioning score of the interaction and if a pharmacist was involved in the interaction were significant predictor of appropriate outcome (p < 0.001 and p < 0.01 respectively). Statistically significant differences were found between median questioning and total scores of interactions involving a pharmacist compared to those that did not (p < 0.001). No statistically significant correlation was found between the number of visits and appropriate outcome (p > 0.05). Conclusions Mystery shopping with feedback did not improve pharmacy staff performance over time. Increased questioning and involvement of a pharmacist in the interaction were significant predictors of referral to a medical practitioner occurring.