ComfyUI-Prediction
ComfyUI-Prediction
Fully customizable Classifier Free Guidance for ComfyUI.
Copyright 2024 by @RedHotTensors and released by Project RedRocket.
Installation
Clone this repo into `ComfyUI/custom_nodes` or use ComfyUI-Manager.
(Optional) If you want beautiful teal PREDICTION edges like the example apply patches/colorPalette.js.patch to `ComfyUI/web/extensions/core/colorPalette.js`.
Usage
All custom nodes are provided under Add Node > sampling > prediction. An example workflow is in `examples/avoid_and_erase.json`.
Follow these steps for fully custom prediction:
Some utility nodes for manipulating sigams are provided under Add Node > sampling > custom_sampling > sigmas.
Predictors
Primitive Nodes
All other predictions can be implemented in terms of these nodes. However, it may get a little messy.
Conditioned Prediction
Evaluates your chosen model with a prompt (conditioning). You need to pick a unique conditioning name like “positive”, “negative”, or “empty”.
The names are arbitrary and you can choose any name, but the names may eventually interact with ControlNet if/when it’s implemented.
Combine Predictions
Operates on two predictions. Supports add (+), subtract (-), multiply (*), divide (/), vector projection (proj), vector rejection (oproj), min, and max.
`prediction_A `
Scale Prediction
Linearly scales a prediction.
`prediction * scale`
Switch Predictions
Switches from one prediction to another one based on the timestep sigma. Use sampling > custom_sampling > sigmas > Select Sigmas to create a sub-ranges of timestep sigmas.
`prediction_B when current_sigma in sigmas_B otherwise prediction_A`
Scaled Guidance Prediction
Combines a baseline prediction with a scaled guidance prediction using optional standard deviation rescaling, similar to CFG.
Without `stddev_rescale: baseline + guidance * scale`
With `stddev_rescale`: See §3.4 of this paper. As usual, start out around 0.7 and tune from there.
Characteristic Guidance Prediction
Combines an unconditioned (or negative) prediction with a desired, conditioned (or positive) prediction using a characteristic correction.
`cond`: Desired conditioned or positive prediction. This prediction should not be independent of the unconditioned prediction.
`uncond`: Unconditioned or negative prediction.
`fallback`: Optional prediction to use in the case of non-convergence. Defaults to vanilla CFG if not connected.
`guidance_scale`: Scale of the independent conditioned prediction, like vanilla CFG.
`history`: Number of prior states to retain for Anderson acceleration. Generally improves convergence speed but may introduce instabilities. Setting to 1 disables Anderson acceleration, which will require a larger max_steps and smaller log_step_size to converge.
`log_step_size`: log10 learning rate. Higher values improve convergence speed but will introduce instabilities.
`log_tolerance`: log10 convergence tolerance. Higher values improve convergence speed but will introduce artifacts.
`keep_tolerance`: A multiplier greater than 1 relaxes the tolerance requirement on the final step, returning a mostly-converged result instead of using the fallback.
`reuse_scale`: A multiplier greater than 0 retains a portion of the prior correction term between samples. May improve convergence speed and consistency, but may also introduce instabilities. Use with caution.
`max_steps`: Maximum number of optimizer steps before giving up and using the fallback prediction.
`precondition_gradients`: Precondition gradients during Anderson acceleration. This is strongly recommended, but may decrease result quality in specific cases where the gradients are reliably well-conditioned.
This node is extremely expensive to evaluate. It requires four full model evaluations plus two full evaluations for each optimizer step required.
It’s recommended that you use the Switch Predictions node to skip CHG on the first timestep as convergence is unlikely and to disable it towards the end of sampling as it has marginal effect.
Prebuilt Nodes
Switch Early/Middle/Late Predictions
Convienence node similar to a three-way Switch Predictions node, where the number of `early_steps and late_steps` can be specified directly.
The whole applicable `sigmas` schedule should be provided to the node, and the schedule must be unambiguous when split. (Restart schedules can often be ambiguous.)
Interpolate Predictions
Linearly interpolates two predictions.
`prediction_A * (1.0 - scale_B) + prediction_B * scale_B`
CFG Prediction
Vanilla Classifier Free Guidance (CFG) with a positive prompt and a negative/empty prompt. Does not support CFG rescale.
`(positive - negative) * cfg_scale + negative`
Perp-Neg Prediction
Implements https://arxiv.org/abs/2304.04968.
`pos_ind = positive - empty; neg_ind = negative - empty`
`(pos_ind - (neg_ind oproj pos_ind) * neg_scale) * cfg_scale + empty`
Avoid and Erase Prediction (v2)
Implements a modification of the Perp-Neg algorithm that avoids moving in the direction of the negative guidance.
The result is a guidance vector that should be recombined with the empty prediction using the Scaled Guidance Prediction node or addition in preparation for Characteristic Guidance Prediction.
`pos_ind = positive - empty; neg_ind = negative - empty`
`pos_ind oproj neg_ind - (neg_ind oproj pos_ind) * erase_scale`
Sigma Utilities
These utility nodes are provided for manipulating sigmas for use with the Switch Predictions node. These nodes are under sampling > custom_sampling > sigmas.
One important thing to keep in mind is that ComfyUI SIGMAS lists contain the ending output sigma, even though the model is not evaulated at this sigma.
So, a 30-step full denoising schedule actually contains 31 sigmas (indexed 0 to 30), with the final sigma being `0.0`.
Select Sigmas
This node is an alternative to the Split Sigmas built-in node. It allows you to specify the specific `sigmas to select` by 0-based index as a comma seperated list.
Ranges are supported with `[start]:[end]` syntax. The ending bound is exclusive and negative indices are supported, which index from the end of the list. Both start and/or end may be ommitted.
As a convienience, instead of specifing a list, you may specify `mod ` to select every N’th sigma. You can use this to easily alternate between prediction strageies.
If `chained is disabled, the final sigma in the input list will be dropped. If you're chaning Select Sigmas nodes, you should enable chained` in almost all cases.
If a specified index is out of range, it is ignored.
Examples, assuming a 10-timestep schedule with sigmas `10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0`:
select: 0, 1, 2 chained: false => 10, 9, 8
select: 0, 1, 3:6, -2 chained: false => 10, 9, 7, 6, 5, 2
select: mod 2 chained: false => 9, 7, 5, 3, 1
select: mod 3 chained: false => 8, 5, 2
select: 100, 0, -100 chained: false => 10
select: 5: chained: false => 5, 4, 3, 2, 1
select: :5 chained: false => 10, 9, 8, 7, 6
select: : chained: false => 10, 9, 8, 7, 6, 5, 4, 3, 2, 1
select: -1 chained: false => 1
select: -1 chained: true => 0
select: -1:-4 chained: true => 0, 1, 2Split At Sigma
Similar to the built in Split Sigmas node, this node splits a list of sigmas based on the *value* of the sigma instead of the index.
This node takess a monotonically decreasing list of `sigmas, and splits at the earliest sigma which is less than or equal to the specified sigma`.
This node is primarily useful in setting up refiner models without having to reverse engineer the timestep schedule for differing numbers of steps.
Log Sigmas
Prints the provided list of `sigmas to the text console, with an optional message` prefix to differentiate nodes. This is useful for debugging workflows.
If the sigmas list and message don’t change, ComfyUI will not evaluate the node and so nothing will be printed.
Limitations
ControlNet is not supported at this time.
Regional prompting may work but is totally untested.
Any other advanced features affecting conditioning are not likely to work.
License
The license is the same as ComfyUI, GPL 3.0.