Abstract In the paper, we introduce and characterize a new class of nonconvex functions in mathematical programming, named γ-preinvex functions, which are called γ-invex functions in the case of differentiability. The class of γ-reinvex functions is wider than that of preinvex functions and thus, in the case of differentiability, than the class of invex functions. Some optimality results are obtained under γ-preinvexity assumption for a constrained optimization problem with not necessarily differentiable functions. Further, a number of sufficiency conditions and Wolfe duality theorems are established under γ-invexity assumption. An alternative approach with a modified γ-invexity notion to optimality conditions and duality results is also considered.