Visual Effects Pipelines: A Defaulting Debt
The VFX technological achievements were recognized on February 7, 2017 at the Oscar's 89th Annual Scientific & Technical Awards. The results painted a picture of a thriving industry that is making technical leaps and bounds to sustain itself. As I reviewed the list of the awarded achievements over my morning coffee, I felt a ping of disappointment. What struck me that brisk Vancouver morning, in spite of the joy of seeing my colleagues names in an oscar panel, was the realization as each year progressed, the moving goal line became further and further away. Like a luxury car, VFX studios added all the bells and whistles to our movie making pipelines to make things nice and glossy to sell to clients. Its part of the game; A big pretty demo reel with a lot of cool technology equals big projects and big dollars. As they have added new tools, services, and applications to the large luxury pipeline, they have forgotten to upgrade its engine and safety systems. Eventually the car will not be able to function and fail.
New technology, while amazing achievements, hide away a deeper issue in technology departments for many visual effects studios. Our pipelines are slow, bulky, and are rigid to change and requires large amounts of development to keep up with the goal line. This is from neglecting the core architecture and this neglect eventually builds up like a snowball.
Some refer to this build up of neglect as 'technology debt' or 'tech debt'. And with each new addition we neglect the fundamental pipeline and the debt grows and compounds. Eventually, the debt grows so large that any change/addition to the system includes major overheads to implement. Even as faster technology is added, the subsequent growth of the maintenance and implementation eventually removes any significant gain possible from the addition.
A few years ago this would have been tough to see. Hardware and Software were accelerating in ways that vastly overwhelmed any performance loss due maintenance and implementation. Now, computers are not getting that much faster and thus the technologies industries have turned to distributed systems to get that leaps in performance they need to maintain a competitive edge. VFX studios, in turn, tried to follow. This simple shift has revealed a 20 year old growing tumour on the side of some VFX pipelines. Poor Architecture.
Creating and Implementing Scalable Systems is tough and costly and requires a healthy respect for the future and development. For technology, the future is always bitter sweet. As we get the benefits of new tech we also have to deal with the pain of refactoring old systems to work with the new tech. For VFX, this has become a staple of development that is skipped. Some new curve jumping patented open source technology framework came in and it needed to be implemented yesterday. Next thing you know, it is all hands on deck to get the artists using it. Meanwhile, no one has considered the consequences of the addition. There is no arguing for the pipeline's architecture and whether we have implemented this correctly. If there is a defender, then the phrase synonymous with VFX is heard "We just need this working, we can come back to it later". We never go back......
How did you get here?
A common trend in visual effect facilities is to have an army of artists and a few developers. Developers are tough to get executives to pay for. They are expensive, their results are not instant, and their benefits seem invisible until the systems they maintain break. What has happened is the natural evolution of artists to learn to program in order to meet the needs of a production. For a supervisor, having an artist that can both make her tools and use them is something of a gold mine. Overtime these technical artists started making larger tools and VFX studios begin to rely more heavily on artist developers.
After some time, artists have made little tools which got used together to create a pipeline and soon entire systems relied on tools built by naive artists just trying to do their job. Savvy and well versed in a language and software, they had little knowledge of software engineering. In the future you began to notice making small changes to you system was tougher. Replacing a core component, like a render service, suddenly became a monolith project. You realize suddenly that the system that worked so well is suddenly impossible to maintain. You just defaulted on your tech debt.
Spotting tech debt
Defaulting on tech debt can have many signs, here are some I have seen:
- Developers appear to be overworked
- Minor additions and fixes to the system cause errors in other seemingly unrelated areas of the pipeline.
- Replacing a component requires changes to artist facing code.
- Systems begin to exhibit instability under light loads.
- Testing the pipeline is nearly impossible.
- Getting information about the systems status is a joke
- For each layer of your pipeline its tough to find a major component you could replace with minimal effort.
Defaulting a your tech debt is tragic and can sometimes be too much to be able to remedy it. I have seen companies fail by approaching the debt without the guidance of good architects. The story roughly goes like this each time...
Artists and Managers are complaining about issues with the pipeline and how the system is not working, failing constantly, or difficult to use. Managers and Executives assemble a meeting and discuss how to fix this problem because its affecting profits. After a few rounds of finger pointing they decide the studio needs to rebuild the pipeline from the ground up or implement some amazing tool to solve all our problems. Managers will come up with a technology map that will outline everything they need and a timeline to do so. Executives allocate the money and the technology accepts it because they never have had the chance to do this type of creative work. The rebuild starts. It never ends and the next upgrade is always a few months away. Money is pouring out for developers that are always trying to finish that next thing. When we try to rebuild the plane as we are flying, its always going to crash.
It will never cease to amaze me how many times I have heard or seen this pattern. Companies tend to think in terms of dollars and teams, not software architecture and agility. But do not be mislead, they want those things. Its tough to get an artistic supervisor to understand why the code is more important then the frames sometimes and their are occasions the frames are more important. We have to, as developers, teach people that incremental improvements and good architecture can save you from rebuilds. Rebuilds are costly and rarely accomplish what the say they. Introducing the new shiny shotgun toolkit, integrating a fancy Universal Scene Description pipeline (USD), or using the latest and greatest renderer is not fixing your bad architecture. It's making it worse. Relying on out of the box tools to fix you bad architecture, while convenient, creates a new type of tech debt that is more costly later.
Genius software engineers understand that good architecture (with direction) will allow you to use your fancy new tools but until you can map the architecture of your major components for the pipeline on a few sheets of paper, you are no step closer to relieving your tech debt or preventing it from growing. Tech debt is not something you can pay off all at once. Like real debt, you have to chip at the debt piece by piece.
In pipelines, that means identifying core components, locating duplication, merging like components, abstract and decouple the code, refactor the code into a new form and replace with a better architecture. For very large monolith systems, it is like playing pick-up-sticks where each stick is a component you have decoupled/abstracted. You continue to iterate like this until the architecture of your system is better and then you keep doing that. Whenever a new addition is added, before you blindly add it in, ask how does this fit into the architecture? This takes discipline and some creativity, luckily you are working with some of the most creative people in the world. Prioritizing a stable system means you can spend less time fixing it and more time developing it.
Why should you invest in good architecture?
Never sacrifice good architecture. Good architecture will make you money while poor architecture will cost you double. Not every pipeline is perfect but breaking the cycle of poor architecture will make developing far more enjoyable for your developers. In turn this keeps your developers in house and maintaining the pipeline. Good architecture frees up a developer's time. Idle developers are your greatest asset for the future. An idle developer will typically start developing piece of the pipeline they know needs some love or come up with new tools that add overall improvements to the ecosystem that managers did not see. That will land you that next Oscar. So ask yourself, do you have good architecture?