This paper describes a work in progress on co-evolving Artificial Neural Networks (ANNs) for music improvisation. Using this neuro-evolutionary approach the ANNs adapt to the changes in the human player's music as input, while still maintaining some of the structure of the musical piece previously evolved. The system is called PRIMAL-IMPROV and evolves modules that are composed of two ANNs, one controlling pitch and one controlling rhythm. The results of a quantitative study show that, by only introducing simple rules as fitness functions, the system is able to generate more interesting arrangements than ANNs evolved without a specific objective. The emerging and interesting musical patterns that are produced by the evolved ANNs hint at the promising potential of the system.