The very same problem happens on XFX Fury X and 15.7 driver (or later, not sure, don't have it here). The situation was even worse in Windows 8.1 with the taskbar: non-primary display is forced to use scaling of the primary display on the desktop. Windows 10 renders taskbar correctly, bit icons still keep primary-display-scaling on all display so, when 5K is at 200% and primary, other 100% scaled displays keep huge icons, 2x in size. That looks ridiculous:
What you just described is a different issue. Yes, scaling fails on non-supportive apps when transitioning from display to display, but, I find that a minor problem and, as you say, app problem, not Windows. When I open Skype for Business while having 5K as primary, it is huge if I move it to any other display. But that is also a minor annoyance that can be fixed with an application update.
I find rendering clarity a more serious issue. What is in that screenshot up there is actually a clean start: changing primary setup, log out for changes to apply and run the application again (Firefox, Chrome, Blender, Visual Studio whatever). All applications have the same issue, including Visual Studio.
Try running a more extreme setup, a 200% scale. I presume you can't see the difference because of a combined effect of a larger dot pitch and smaller scale difference. On a larger scale it is really poking eyes, especially a problem with already a not-so-sharp "normal" display not being primary. Normal displays already have lower sharpness and then this issue.
Yes, someone who uses 2 or 3 identical 4K/5K displays does not really care since scaling is identical. But those who mix do have a problem that is really annoying. It is even easier to make lower-resolution display primary, since a high-resolution one would look less-worse, but that completely beats the purpose of scaling and high-res in the first place.