A robust and reliable algorithm for computationally efficient design optimization of microwave structures evaluated with electromagnetic (EM) simulations is introduced. The presented approach utilizes a low-fidelity model constructed from coarse-discretization EM simulation of the structure under consideration, as well as cheap derivative information obtained by means of adjoint sensitivity. Adjoint sensitivity is exploited to reduce the misalignment between the low- and high-fidelity models through input space mapping, to construct a first-order consistent surrogate model through manifold mapping, and to optimize the surrogate through a trust-region-based algorithm. This comprehensive use of adjoint sensitivity results in a very low design cost and excellent convergence capability of the proposed algorithm. Efficiency of our approach is demonstrated using two examples: an ultra-wideband antenna and a microstrip bandpass filter.