The dynamics of desorption from a submonolayer of adsorbed atoms or ions are significantly influenced by the absence or presence of lateral diffusion of the adsorbed particles. When diffusion is present, the adsorbate configuration is simultaneously changed by two distinct processes, proceeding in parallel: adsorption/desorption, which changes the total adsorbate coverage, and lateral diffusion, which is coverage conserving. Inspired by experimental results, we here study the effects of these competing processes by kinetic Monte Carlo simulations of a simple lattice-gas model. In order to untangle the various effects, we perform large-scale simulations, in which we monitor coverage, correlation length, and cluster-size distributions, as well as the behavior of representative individual clusters, during desorption. For each initial adsorbate configuration, we perform multiple, independent simulations, without and with diffusion, respectively. We find that, compared to desorption without diffusion, the coverage-conserving diffusion process produces two competing effects: a retardation of the desorption rate, which is associated with a coarsening of the adsorbate configuration, and an acceleration due to desorption of monomers "evaporated" from the cluster perimeters. The balance between these two effects is governed by the structure of the adsorbate layer at the beginning of the desorption process. Deceleration and coarsening are predominant for configurations dominated by monomers and small clusters, while acceleration is predominant for configurations dominated by large clusters.