< Previous | Contents | Manuals Home | Boris FX | Next >
Solving Long Shots
SynthEyes can be used to solve shots containing thousands, even tens of thousands, of frames and trackers, with sufficient skill, effort, CPU cycles, and RAM. Algorithm improvements in SynthEyes made solving long shots much more practical.
Note : Depending on your needs, solving a shot in pieces may be an efficient and practical approach, for example for 360VR stabilization. See the tutorial on Solving Long Shots in Pieces and the Splice Paths tool.
The viability of processing long shots depends on the quality of the tracking data being presented. If the tracking data is good, SynthEyes will proceed through the shot at a rate determined by the number of frames and trackers, so it's mainly a question of how long it takes. (You'll want a fast processor and lots of RAM: 32 GB is a starting point at present.)
You can accelerate the process by having SynthEyes initially solve only every second or third frame using the Decimate control on the Advanced Solving panel. You could choose a larger value, but it is a trade-off based on how long trackers last—a tracker must be valid on a reasonable number of decimated frames to be able to contribute, so if the decimation value is too high, trackers will only be valid on one or two frames and the solve will fail. You can use larger values if the shot is very slow moving and trackers last a long time.
After you get an initial solve using decimation, you can set the decimation back to one (ie none, using every frame), set the solving mode to Refine, and have SynthEyes solve for the intervening frames as well.
If the tracking data contains notably bad trackers, ie trackers on actors, moving vehicles, spurious lens flares, etc, the solve can be thrown completely off, causing the RMS hpix error to start increasing to very large values, and the solver to encounter numerical problems (like dividing by zero, but different) that make progress impossible. Long shots are virtually always done with automatic tracking, not supervised, and the automatic tracker can put trackers on problematic features.
For that reason, automatic tracks always need to be cleaned up manually, but that can be difficult on very long shots, where even playing through the shot at full playback speed requires significant time, let alone examining it slowly enough to visually detect problematic trackers.
The Find Erratic Trackers tool can help with this, but it's accuracy is subject to the quality of the imagery. Unfortunately, shots with significant lens distortion, rolling shutter, or 360VR footage containing stitching problems is unsuited to automatically detecting problematic trackers.
As a result of all this, long solves always run the risk of developing problems during the solve. You don't want to have to repeat a multi-hour solve from scratch several times. The challenge is to identify problems immediately, so you, the tracking artist, can identify and correct them, and then be able to resume the solve.
SynthEyes gives you the tools to do that via the Advanced Solver Settings panel ('more' next to Go! on the solver panel ), which we'll discuss here. You also have the ability to adjust some solver parameters to optimize the rate at which your solve can be processed, and minimize the chances the solve will encounter serious problems. We'll start with that first.
On long shots, we recommend setting If looks far to Make ZWT, because as the camera moves forward, trackers that initially looked far away may become very nearby. They must not transition to far, and they can be trouble while they are initially far away, so it's best to disable them (assuming you have many trackers per frame, as you
should). The Make ZWT option not only disables them, so they don't cause trouble now, but makes them zero-weighted trackers (ZWTs), preventing them from later reviving as zombie trackers and destroying subsequent Refine solves. Though some trackers that are initially disabled can actually later be successfully revived, it is better to make them ZWTs and examine and correct or delete them on a case-by-case basis, with no chance of their becoming destructive zombies.
As the solver is working through the shot, it is alternating between adding more frames to the beginning or end of the set of solved frames (ie camera or moving object position or orientation), and adding more trackers to the solve. After adding trackers or frames, it then iterates to find the best solution, before adding more frames or trackers. (Simplified for this discussion.)
When frames or trackers are added to the solve, they arrive with initial estimated positions. By its nature, an estimate will ultimately be shown to be wrong, ie contain some error. Therefore when you add frames or trackers to the solve, you are adding a lump of error. If that lump is too large compared to the solve, the solve will fall apart.
To limit the amount of error added to the solve at a time, you can reduce the Max frames/pass and Max points/pass setting. Reducing either or both settings will reduce the error added, typically reducing the iteration count (more speed), but increasing the number of passes required to add all the frames or trackers (less speed). So it's a trade- off. For noisy 360VR shots, reducing the Max frames/pass value is most beneficial.
When trackers or frames are added, the solver then iterates to find the best solution, ie that with the least error. Each iteration takes the current solution and makes it a little better, until there's nowhere better to go (it "converges"). Most of the time that happens fairly quickly, a few tens of iterations. Sometimes that doesn't happen, and the solver keeps making tiny corrections. There's a limit (Max iterations/pass) on how many iterations will be performed before we decide that's dumb.
When the iteration count limit is hit, we need to decide what to do. If the Stop on iterations checkbox is checked, the solver just stops so you can look at what's happening immediately. Alternatively, we can just plow on, adding more trackers or frames, and hope that what we have is good enough, and that the additional data will help decide what's happening. Plowing on usually works.
The Max iterations/pass limit gives you control over when this happens. On big long solves, it can be worthwhile to lower the limit down to 80, say, so that we just move on to the next part of the solve without doing a lot more slow, senseless, iterations.
When a bad tracker or two does appear, typically what you'll see is that the overall error once it has converged suddenly increases. (Errors are much higher while iterating). You might be running along with an error around one pixel, then some frames or trackers are added, and the best error starts going up into the five or ten pixel range or higher. (It's not necessarily immediate.)
You can cause the solve to stop when this happens by setting the Stop if error exceeds value. The default value, 15, is intentionally high; values down near 3 might be more appropriate. Once stopped, you can repair and restart (more discussion below).
Bad trackers can also result in numerical issues in the solver ("Warning: solution not crisp"). As with the iteration count limit, you can decide what you want to do when this happens, either Stop, press ahead as if it converged (Advance and Hope), or Proceed Slowly, which uses a different algorithm which skirts the numerical issues, but is inherently and unavoidably incredibly much slower.
If you're trying to run a long solve in the background overnight, say, you might try the Advance and hope approach, otherwise Stop is recommended. The Proceed Slowly option is probably reasonable only for smaller solves, maybe a few thousand frames and maybe a thousand trackers. It is used for the first ten seconds of every solve though!
Once a long solve has stopped without completing, due to an iteration limit, hpix error limit, or trouble mode setting, you then need to repair the situation and restart the solve.
To identify the problem area, begin by looking at the solver output. Each pass, you'll see the range of frames that have been recently added to the solve (possibly at both beginning and end). You'll want to look for bad trackers in the general area of the most recent frame range (or the range where the error started to degrade substantially).
You'll typically want to adjust the playback range (little green and red triangular markers in the time bar) to the general problem area, and let the shot prefetch fill in all the frames in that area, so you can scrub and play through it efficiently and hunt down the bad tracker(s). The time bar's right-click menu, especially Set playback range to visible, is helpful in doing this.
The most-recently-added trackers are automatically selected when the solve stops before completing. (There's a preference to disable that.) There are many features to help you find bad trackers (most in the camera view):
the camera view's right-click/View/Show tracker radar,
turning off image display via right-click/View/Show Image,
the View/Show 3-D only if visible,
the camera view's right-click/View/Only selected trackers option,
the ability to group the trackers by assigning colors on the Tracker panel (a little easier to work with than viewing only selected trackers),
the Error sort of the Graph Editor or main window to identify the worst trackers. The tracker with the worst error is NOT necessarily the one causing the problem, though! You'll have to look around.
Typically most very-bad trackers will just be deleted, though in some cases you might use the graph editor to split a tracker if it has switched from one feature to another and is particularly valuable.
Once you've corrected the trackers, you're ready to restart the solve using Refine mode. But don't do that quite yet!
The reason is that the previous, now removed, bad tracking data has already corrupted the solve so much that it set off the alarms. You don't want to continue using it, nor do you want to start over.
Instead, click Unsolve frames on the Advanced Solver Settings panel. Tell it to unsolve a hundred or few hundred frames, depending on how long it took for the problem to be detected and stop the solve. You want to unsolve past that point, since when new tracker data is introduced, it affects existing solved frames as well. But its impact on the camera/object path lessens the further away from the tracker's lifetime it gets. You want to unsolve not only past the bad tracker(s) but past the region they had significant impact on as well.
Again, you'll need to look back into the history of frames being added to determine whether to unsolve from the beginning or end (or both) of the solved section of the shot.
Note that the Unsolve frames tool also unsolves any trackers that depended on the range of frames that are being unsolved. If enough trackers are being unsolved, and there aren't that many, this can result in some extra frames being unsolved as well.
You should adjust the Minimum frames and Minimum trackers values, based on the amount of continuity and number of overall trackers, to determine at what point under-supported frames and trackers are unsolved. This is a subjective decision. Lower numbers are necessary when there are a minimal number of trackers but result in too few problematic results being unsolved. Larger numbers ensure that the remaining portions of the solve are solid but run the risk of unsolving large portions of the shot— possibly unsolving the whole shot!
You have the option to have the tool unsolve only from the end, or also check the whole shot. Checking the whole shot is recommended. Checking the whole shot may also be necessary if the solver stops and tells you to do so, if you’ve deleted too many trackers from a solve during your cleanup process.
There is a risk, especially with higher minimum-frames or -trackers settings, that the solve will become disconnected, separated into independent solved sections with no solved trackers in common. Unsolve does some simple checking for this (only for non- survey shots) but a complete analysis isn’t possible in general. The solver will not be able to proceed, and you’ll ultimately need to unsolve an entire section by selecting the Playback range option on Unsolve, and setting the minimum values to be large. So it’s worthwhile not to have Unsolve do too much unsolving.
The Unsolve frames tool sets the solving mode to Refine (or Refine Tripod), so you're then ready to hit Go! again and continue the solve.
The better your initial cleanup of the tracking data, the fewer times the solver will have to stop for your attention!
Tip : If the solver's Constrain checkbox is turned on for long shots, avoid Distance constraints between trackers from much different parts of the shot,
as that may degrade performance. Use *3 constraints or Place mode (many locked points) instead.
©2024 Boris FX, Inc. — UNOFFICIAL — Converted from original PDF.