Stock market prediction is extremely important for investors because knowing the future trend of stock prices will reduce the risk of investing capital for profit. Therefore, seeking an accurate, fast, and effective approach to identify the stock market movement is of great practical significance. This study proposes a novel turning point prediction method for the time series analysis of stock price. Through the chaos theory analysis and application, we put forward a new modeling approach for the nonlinear dynamic system. The turning indicator of time series is computed firstly; then, by applying the RVFL-GMDH model, we perform the turning point prediction of the stock price, which is based on the fractal characteristic of a strange attractor with an infinite self-similar structure. The experimental findings confirm the efficacy of the proposed procedure and have become successful for the intelligent decision support of the stock trading strategy. The online version contains supplementary material available at 10.1007/s10115-021-01602-3. © The Author(s), under exclusive licence to Springer-Verlag London Ltd., part of Springer Nature 2021.