Troubleshooting is an essential skill in any part of the sound world and especially on tour. On tour, you’re often on a tight schedule, in an unfamiliar city, and outside a quick delivery distance of the shop. Learning to diagnose and fix (or work around) problems on your own is the only way to keep a show running.
Of course, it’s also one of the hardest things to learn because there are an infinite number of problems that can pop up and most of the time you figure them out as they come your way, building a database of issues and fixes for your future self.
However, if you develop a system of how to approach problems, even unfamiliar gear or new gremlins become progressively easier to solve.
First, never discount these basics. Turning it off and turning it back on is sometimes the power cycle that’s needed if things didn’t start up in the right sequence or it’s simply checking that the piece of gear is actually plugged in. I once spent 15 minutes trying to figure out why a keyboard wasn’t transmitting MIDI (reseated and swapped out cables, got a new MLA (MIDI Line Amplifier), and more) only to realize that the keyboard itself had been off the entire time.
After the basics, you follow the signal flow. Learning how a system goes together is key to fixing a problem. How does signal get from Point A to Point Z and what are the patch points, cables, adapters, and everything in between? Connection points at adapters or racks are typically the weakest points in any chain, whether something didn’t get plugged in (jokingly referred to as a “high impedance air gap”) or strain on a connector that results in a band connection.
For example, some of our cameras on tour are built into the lighting trusses that hang over the stage. From the camera, there’s a cable that goes to a rack that lives with the dimmer racks (we call it “Dimmer Distro” on our tour) which is a distribution point for all the cameras and speakers that live on truss or lighting ladders that Electrics sets up during load in. At the Dimmer Distro rack, the cable patches into a panel which barrels through to a repeater inside the rack. The outputs of the repeater connect to another panel where our Crossover (XO) bundles plugin to get the camera signal from the Dimmer Distro rack across the stage to Ampland. In Ampland, there’s another panel that connects the XO run to the input of a video matrix.
As you can see, there are a lot of sections to troubleshoot if something goes wrong. And that’s just getting the camera to the central hub. There’s a whole other set of cables and connection points to get the video from the matrix out to the SM call desk or Automation or FOH or any number of other places. Learning the signal path of each of the systems in your production (audio, video, com, etc) will make it much easier to eliminate possible causes as you work through troubleshooting logic.
Ask yourself, what is the problem? Be as specific as possible. “Video is flickering” and “the FOH color shot at the automation rack is intermittently going out” are two completely different problems.
Next, ask questions to narrow down the problem. Learning the right things to ask can significantly narrow down what you have to troubleshoot.
Continuing on the video theme (com and video will be the things you have to troubleshoot most often), let’s run with the problem of the FOH color shot going in and out at automation. First question: Is the FOH shot doing the same thing everywhere? Usually, there will be multiple places that have the same video. If the FOH shot is stable in other locations (Fly Rail, SM desk, etc) that means the video input itself is okay, there’s just some problem getting to Automation. If it’s flickering everywhere, the issue is more likely on the input end, somewhere getting from the camera to the video matrix. But, by answering that one question, we’ve eliminated about half of the system (either the output side, if it’s just automation, or the input side if the problem is in multiple places) that we need to look at.
This is also something you ask when there is a main and backup (or spare) in the system. Does one work but not the other, or is the issue the same for both?
Our keyboards have a main and backup system, so when there was signal loss on one of the channels, we tested the backup as well. When that also had a problem that told me that the issue was somewhere after the two inputs merged. It ended up being a bad line coming from the pit rack up to our DigiRack in Ampland and checking the backup helped quickly eliminate a section of the system so we could move on to more likely culprits.
In this case, again, I looked at signal flow. The keyboard is sending MIDI messages to two computers: one Main and one Back Up. Those computers each send three stereo pairs to an interface, and that interface sends to a switcher that takes the two inputs and (via a pedal at the keyboards) can switch between the two signals to send a single set of three stereo pairs of audio from the pit up to the DigiRack in Ampland (with panel inputs at each rack). Then the DigiRack connects to the console via a fiber loop.
So, first, we identified the problem: “The Left channel of Key 2A (again 3 stereo pairs: A, B, and C) is coming into the board at a significantly lower level than the Right.”
Next, narrow it down:
“If we swap to the Backup, does that fix it?” Answer: No. The problem is somewhere after the interfaces connect to the switcher.
“Is it happening anywhere else?” Answer: No. Likely it’s not the fiber link or there would be other channels having the same issue.
This eliminates the first couple of links in the chain and the last one. The problem is likely either the run between Ampland and the Pit, the tails on either end or the card in the DigiRack.
Now you can start to change things. (As a note, it’s best to change one thing at a time, check if that helped, restore, then change something else. If you change a whole bunch of things and it starts working, you won’t actually know what was the problem.) In this case, I swapped the line for Key 2’s A-Left channel to a spare line in the run to Ampland and that got the channel back to a good level.
Conclusion: Bad line on that mult or one of the tails. (On tour, fix the problem, then you can do more specific tests when you have a work call or more time.)
Consistent problems like this are usually easier to identify and solve: you know when you’ve fixed the problem because the problem goes away. With intermittent issues, it’s the opposite: you can identify what the problem is only when you can recreate it, or you have to wait and watch for a while to verify that it’s not coming back.
For these problems, still, try to eliminate as much of the system as you can so you can focus on the most likely fixes. Then, find out how to recreate the problem. Pre-Covid, I had a bad fiber line that was causing pops in the system: it would be consistent popping when the system was first up, but after about 30 minutes the pops would be intermittent or stop altogether. At the console, I could see that it was happening over multiple channels, all of which were from one of the DigiRacks in Ampland. That told me that it was likely an issue with the fiber loop that connects the racks to the console. We have seven fiber links that create our loop (the console, three DigiRacks, one Mini Rack, and an SDRE), so we spent a work call powering up the system, unpatching one link at a time, and waiting to see if there would be another pop. If there was, we’d re-patch and unpatch the next link in the loop. Sometimes the pops would go away on their own and we’d have to power down and power back up to see if we’d actually solved the problem, or if it had just shifted from consistent to intermittent popping. It probably took at least three or four rounds through to finally identify the bad line. Once we were fairly certain we had the right link, we ran a spare and sent the bad fiber reel back to the shop for repair. However, after several shows with pops at random intervals, it was a few more days before I was able to relax from being on high alert and trust that we’d actually fixed the problem.
Sometimes you don’t have time to solve the problem, just put on a band-aid until you can get a work call or a point in the show where you can actually take a look at what’s happening. This is why you have backups and spares. On Les Mis, the backup engine on the console crashed mid-show and we ran without it for a couple of shows until the shop could send out another engine. If someone has a midshow mic pop, sometimes it’s easier to toss a new rig on them and take a look at the old mic back in Ampland instead of trying to diagnose an issue during a 30-second quick change. Recently on Mean Girls, we had one of our RF antennas disconnect sometime between preset and the show starting. Since we couldn’t get out to the towers where the antennas were mounted during the show, my A2 set up the spare antenna backstage so we had something to work with until we could check it out at intermission.
Always remember that you can ask for help. You’re only as good as your tools and your network. When my console crashed, I called a DiGiCo tech and then the shop to help me work through the problem. When our keyboards have problems, I ask our keyboard tech if he notices anything I might have missed. When video is being a pain, my A2 and I will tag team on monitors: someone can swap cables while the other one watches the monitor so we know right when a problem acts up again or if it’s fixed.
Testers are also your friend. Signal generators like QBox or CabDriver help you move step by step through the signal chain to check things. Personally, I use my CabDriver to send pink noise down the NL8 lines to my cluster as I’m building it and it’s still within easy reach. Once I’ve heard a signal through the speakers, I know for sure when I fly it out 30+ feet that everything’s functioning from the Ampland end of the bundle to the speakers. Multimeters can help you with electrical connections and cable testers (especially one that supports multiple connectors) are indispensable. Personally, I have an SM Pro CT-3 which has two separate parts so you can split it to take it to two different places (troubleshooting usually means that you don’t have both ends of the cable in the same place) and has indicator lights to tell if something is cross-wired.
Troubleshooting is a skill that takes time and practical application to build. The more problems you see and solve, the better you’ll get. It starts as a slow process, so give yourself some patience and try to work through problems as logically as possible.