We describe a technique that uses tractography to drive the local fiber model estimation. Existing techniques use independent estimation at each voxel so there is no running knowledge of confidence in the estimated model fit. We formulate fiber tracking as recursive estimation: at each step of tracing the fiber, the current estimate is guided by those previous. To do this we perform tractography within a filter framework and use a discrete mixture of Gaussian tensors to model the signal. Starting from a seed point, each fiber is traced to its termination using an unscented Kalman filter to simultaneously fit the local model to the signal and propagate in the most consistent direction. Despite the presence of noise and uncertainty, this provides a causal estimate of the local structure at each point along the fiber. Using two- and three-fiber models we demonstrate in synthetic experiments that this approach significantly improves the angular resolution at crossings and branchings. In vivo experiments confirm the ability to trace through regions known to contain such crossing and branching while providing inherent path regularization.