Abstract When drilling circular openings (e.g., tunnels or boreholes) in brittle rock, the in-situ stress conditions are often such that failure is initiated at or near the wall of the opening. In this work, a mechanism of open hole instability is considered based on growth of pre-existing micro-fractures in the direction of greatest compression. The major factor enabling the pre-existing 3-D cracks to propagate extensively is the presence of the intermediate principal compression near the opening wall (in the direction of the opening axis). The unstable growth of wing cracks leads to separating thin rock plates (flakes) from the bulk of the rock mass followed by their buckling, separation, and exposure of the fresh surface. Then this process of rock surface spalling repeats itself eventually changing the shape of the hole. As the opening develops, its shape becomes elongated which, in turn, can affect this mechanism primarily through continuous changes in the stress concentrations around the opening. The sole cause of the unstable phase of crack propagation is the crack–boundary interaction. The opening develops if the unstable crack growth proceeds at least up to the buckling size. Otherwise, the opening shape gets stabilized. This approach also allows for determining the final stable cross-section, as well as its relationship to the applied boundary stresses. The extent of failure is primarily determined by the initial parameters of micro-crack distribution.