Abstract Complex permittivity and permeability of a magneto-dielectric composite material is studied using shielded, conductor-backed coplanar waveguide technique over X-band. The propagation constant over the test frequency range is evaluated using element-to-element correspondence of the ABCD matrix computed from scalar scattering parameters without modifying the coplanar structure. The mathematical formulation is verified by performing permittivity and permeability measurements for air over the range 9.5–12.045GHz. Nano-sized nickel ferrite particles was synthesized using co-precipitation technique at different sintering temperatures. Average crystallite diameter is found to be 6.63–17.55nm. The composite with 2% and 4% volume fractions is fabricated by reinforcing 6.63nm ferrite particles in low density polyethylene matrix. The complex permittivity and permeability of the composite obtained from this method are also verified with cavity perturbation technique. The composite shows low saturation magnetization of 1.8745emu/g and reduced hysteresis loss at room temperature.