You can absolutely upgrade from midfi to hifi tier and not like the result. There are matters of synergy and taste at play. Look at the used market! Lots of high end gear getting constantly shuffled around. Lots of mid and low tier gear too, and some of those sellers may be trying to reach for the next tier for satisfaction :)
However, higher tier components get to remove budget constraints in their design and implementation. This allows changes that matter....and some that probably don’t (cosmetics). But IF it was engineered well and voiced right and synergizes well with your system...it can be magic. If you get one of these components to elevate your system, it will probably hype you up to go buy more high-tier components...which you may experience mixed results with :P