We develop an agent-based model of the motion and pattern formation of vesicles. These intracellular particles can be found in four different modes of (undirected and directed) motion and can fuse with other vesicles. While the size of vesicles follows a log-normal distribution that changes over time due to fusion processes, their spatial distribution gives rise to distinct patterns. Their occurrence depends on the concentration of proteins which are synthesized based on the transcriptional activities of some genes. Hence, differences in these spatio-temporal vesicle patterns allow indirect conclusions about the (unknown) impact of these genes. By means of agent-based computer simulations we are able to reproduce such patterns on real temporal and spatial scales. Our modeling approach is based on Brownian agents with an internal degree of freedom, θ, that represents the different modes of motion. Conditions inside the cell are modeled by an effective potential that differs for agents dependent on their value θ. Agent’s motion in this effective potential is modeled by an overdampted Langevin equation, changes of θ are modeled as stochastic transitions with values obtained from experiments, and fusion events are modeled as space-dependent stochastic transitions. Our results for the spatio-temporal vesicle patterns can be used for a statistical comparison with experiments. We also derive hypotheses of how the silencing of some genes may affect the intracellular transport, and point to generalizations of the model.