Software Engineering is like all other engineering, but all engineering is an art when it is young.
Imagine the first person to build a stone bridge with a support across a river. That person was probably an artist and visionary of immense talent. Many artists with extreme talent end up in engineering, but even so, almost all the people you encounter in “Engineering” today are simply copiers.
They take components and pieces designed by artists and place them in patterns that are copied from something an artist once made.
In software engineering, we are still creating the components and patterns. This means that without an artist on your team, you are very likely to fail. This will not always be true, but until artists create enough tools for programmers to use, the jobs will be difficult to schedule and results will be bug-ridden and sloppy.
Waterfall works pretty well if you have an artist on your team (which was more common when Waterfall was created, but less so as we moved through the 90’s and the tiny pool of artists diluted.)
Agile development is an acknowledgment that very few teams have artists on them and a set of methodologies to compensate.
Consider the difference between hiring Michaelangelo and having him pick talented young laborers to paint a Chapel roof vs going out yourself and hiring a bunch of house painters and saying “None of us really understand this process, so let’s allow an extra month in the schedule”.
The house painters don’t have a chance, and it’s this fact that causes the 80-95% of software projects that fail.
I guarantee there are at least a few at SUN, that beautiful, clean Java API didn’t write it self. No matter how many times you iterate, without the vision you will not get to beauty, you will just (barely) get to functional.
Also, My opinion is that like any other art, much of your aptitude is set before you are a teen. If you aren’t eating code for breakfast, lunch and dinner after your first 3 years of exposure to a PC, you’ll never be an artist.
Just like if you are not playing decent piano within the first few years of full-time exposure to one, you’ll never be a master. You can be good, you can play with a group, but you are not going to fill a concert hall alone.