800 Carnarvon Street, #323 New Westminster V3M 0G3

Troubleshooting: Not taught enough in animation/vfx schools

Troubleshooting is a critical primary skill in digital media, including 3d animation and visual effects. It is critical because so much can go so wrong so quickly. ‘Primary’ because things often do go wrong and it’s up to us as animation / visual effects artists and technical directors to sort out the problems and fix them. You really can’t work in the digital media without decent analytical and troubleshooting skills. The question must be raised: why do so many people graduate from even the best visual effects schools without decent troubleshooting skills?

So this article is here to help you fix those pesky issues.

Problems pop up for many reasons. There might be a bug in the software preventing it from behaving as advertised. A parameter may have been changed inadvertently. Several different tools may be interacting in an unexpected way. Or the artist / TD may have an incomplete understanding of the tools that culminates in an improper or invalid result.
Problems are often difficult to ferret out. One reason for this is that digital artists and TDs tend to love creating highly complex solutions to simple problems. They often consider it a measure of their technical prowess to show off how exceptionally cool their solutions are, believing that complex solutions are more impressive. But this results in wildly over-engineered shaders, rigs, dynamics, lighting or any of a hundred other things. The fact is that the simplest solutions are always the best. After all, commercial art is a business not a “look how awesome I am” show.

Any crew member can expect to run into many problems per day and may be required to troubleshoot and resolve each and every one before they can continue. The big problem is that so few digital artists conduct a really good trouble-shooting procedure when trying to figure out a problem so they often come to incorrect conclusions and therefore make changes that actually make matters worse rather than fixing them.

Here’s an example I once ran into.
A shot came up for review in dailies one day. This shot needed a minor adjustment to lighting before final render. Specifically, the shadow-side of the CG object was too dark and so the note was given to lift the shadow-side of the object and resubmit the shot for review. The artist responsible chose to lift the shadow side by increasing the constant value of the material, which did appear to solve the problem. However increasing the constant value of a material makes it appear to “glow in the dark”, so from that time forward any time we saw the object in lower light conditions it always appeared too bright and required further adjustment to darken it.
The actual problem had been that the light intensity for the indirect light sources was too low, however the artist, instead of increasing the indirect light intensity, changed the material to an invalid state where it no longer behaved correctly, creating a false impression of having “fixed” the problem for one shot but in fact making matters worse and causing more work for more shots. Imagine you’re at home and a family member asked you to turn on the lights, but instead you chose to paint the walls a brighter colour. Maybe a glow-in-the-dark colour. Yes, it will make the walls appear brighter, but you still haven’t turned up the lights. Instead of identifying and fixing the real problem, the artist chose to just put a Band-Aid over it, covering up the problem instead of resolving it. All well and good to get the problem off that artist’s plate, but it really just passes the problem along to someone else. The artist should be more responsible than that and realize that if the problem exists for him, it exists for everybody who touches that asset and therefore should be properly resolved.

Band-Aiding a problem instead of resolving it is like throwing skeletons into your closet. Sooner or later, and most likely at the worst possible moment, they’re going to come out and bite you in the ass.

Now here’s the good news:

Troubleshooting procedure isn’t complex or difficult. It involves two simple steps. It is critically important that these two simple steps be done in order, however.
1. Identify the location of the problem
2. Identify the nature of the problem.

The artist who “fixed” the shadow side of that object incorrectly did not correctly identify the location of the problem. He presumed that the problem lay in the object shader when it did not. The problem in fact lay elsewhere in a lighting intensity value. Because the artist did not identify the correct location of the problem, he jumped to an incorrect conclusion and, therefore, never fixed the real problem at all.

With nodes emerging strongly as the workflow of choice for most serious artists, troubleshooting becomes incredibly simple. This is mainly because each node can be enabled and disabled one at a time or in groups permitting the location of a problem to be quickly and accurately identified. Identifying the location of the problem is, perhaps, the most difficult step in the troubleshooting process. Because once the location is correctly identified, the nature of the problem is usually fairly easy to pin down.
A digital scene file can be a big, complex monster. Troubleshooting can take a while. One of the reasons so many Band-Aid’s are used is that many artists and TDs think it’s just quicker and simpler to Band-Aid instead of identify and resolve the real problem. Here’s a news flash for ya…it usually isn’t, in the long run.

Here’s another example:
I was once working on a project in which a series of cat faces had to be digitally replaced in order to make them talk. Shortly after we began working on shots, we discovered a gap between the eyeball and eye socket of the cat which was showing up in the final composite. I was asked to investigate. I found that the modeling artist who had created the digital cat heads had mistakenly made the eyeballs too small so that they did not completely fill the eye sockets. I told the production manager that it would take about half a day to take the models off line, fix the issue, and get them back into production. To me, this was the obvious solution, since we had months of production time remaining and hundreds of shots to complete. His response was that we did not have half a day to spare and that we would instead resolve the problem by having the compositing artists hand paint the gaps out, a process that was going to take about an hour per shot.

Let’s do the math here. The real fix would cost one artist 4 hours and be totally resolved today, never to be seen again. The Band-Aid (painting over the gaps) would cost several hundred hours, would continue to plague the crew for months, and would probably cause additional unforeseen problems in the future, which it did. In the end, after we were in crunch mode near delivery, the issue had caused so many problems that we were forced to fix the eyes at a time when we really did not have the resources to spare, all because one person thought a Band-Aid was better than a resolution.

Now there is a sliding scale to the “Resolution vs. Band-Aid” discussion. The earlier you are in your project schedule, the more sense it makes to dig deeply into problems and resolve them properly. The closer you get to final delivery, the more sense it makes to throw on Band-Aids. There are two reasons for this.

1. There is actually less time available later in the project and difficult problems may push you past delivery. This makes a Band-Aid not only desirable but essential.
2. The further along you are in the project schedule, the less impact any solution will have on the overall project. For example, fixing those cat eyes at the very beginning would have saved time on every single shot, culminating in a massive saving. But waiting until the last weeks of production when 80% of the shots were already final and delivered, the fix only saved time on 20% of the shots. If we had been into the last days, the fix may have taken more time than the Band-Aid. The ship would have sailed on that fix, making it no longer efficient or effective.

So do your homework and eat your Wheaties, sailors. Take the time to resolve issues well and early as much as possible. Early, solid resolution of issues is a huge time-saver over the life of a project and well worth the extra hours it takes to do it. This means it reduces budgets, which makes producers happy, and lets you go home on the weekends to see your long-lost family. Remember them?

So don’t forget your troubleshooting Procedure:


And if you are a faculty member of any animation or visual effects schools, please get with the program. The industry needs better analytical skills out there. If you are someone looking for the best visual effects schools, be sure to ask questions about analytical and troubleshooting skills. The most valuable artists and TDs are defined by these skills.

Leave a comment

You must be logged in to post a comment.