Abstract A structural optimization algorithm is developed for geometrically nonlinear three-dimensional trusses subject to displacement, stress and cross-sectional area constraints. The method is obtained by coupling the nonlinear analysis technique with the optimality criteria approach. The nonlinear behaviour of the space truss which was required for the steps of optimality criteria method was obtained by using iterative linear analysis. In each iteration the geometric stiffness matrix is constructed for the deformed structure and compensating load vector is applied to the system in order to adjust the joint displacements. During nonlinear analysis, tension members are loaded up to yield stress and compression members are stressed until their critical limits. The overall loss of elastic stability is checked throughout the steps of algorithm. The member forces resulted at the end of nonlinear analysis are used to obtain the new values of design variables for the next cycle. Number of design examples are presented to demonstrate the application of the algorithm. It is shown that the consideration of nonlinear behaviour of the space trusses in their optimum design makes it possible to achieve further reduction in the overall weight. The other advantage of the algorithm is that it takes into account the realistic behaviour of the structure, without which an optimum design might lead to erroneous result. This is noticed in one of the design example where a tension member changed into a compression one at the end of nonlinear analysis.