Fusion strategies

A FusionConfig pairs a FusionStrategy with a concrete fusion type. The strategy determines when modalities are merged relative to model training.


from mllabiome.ai_space.multimodal import (
    FusionConfig,
    FusionStrategy,
    EarlyFusionType,
    LateFusionType,
    IntermediateFusionType,
    HybridFusionType,
)

Early fusion

Early fusion merges the feature matrices of all modalities before a single model is trained on the combined representation.

`EarlyFusionType`	Behaviour
`CONCAT`	Horizontally concatenates the feature columns from every modality.
`KRONECKER`	Computes the Kronecker product between modality feature vectors. Applies PCA by default to reduce the expanded dimensionality.
`SUM`	Element-wise sum of feature vectors (requires aligned dimensionality).
`CCA`	Canonical Correlation Analysis projection before combining.
`WEIGHTED_CONCAT`	Concatenation with per-modality feature weighting.

Example:


FusionConfig(
    strategy=FusionStrategy.EARLY,
    fusion_type=EarlyFusionType.CONCAT,
)

Late fusion

Late fusion trains separate models on each modality, then combines their predictions into a single output.

`LateFusionType`	Behaviour
`MEAN`	Averages predicted probabilities across modalities.
`MAJORITY`	Majority vote over class predictions.
`MAX`	Takes the maximum prediction across modalities.
`MIN`	Takes the minimum prediction across modalities.
`WEIGHTED_MEAN`	Weights each modality’s predictions by its training-set accuracy.
`STACKING`	Trains a logistic-regression meta-learner on the per-modality prediction stack.
`SOFT_VOTING`	Averages predicted class probabilities, then takes the argmax.
`PRODUCT_OF_EXPERTS`	Multiplies per-modality probabilities (clipped to avoid zeros).
`CONFIDENCE_WEIGHTED`	Weights predictions by per-modality prediction confidence.

Example:


FusionConfig(
    strategy=FusionStrategy.LATE,
    fusion_type=LateFusionType.MEAN,
)

Intermediate fusion

Intermediate fusion extracts learned representations from each modality’s base model and feeds them into a meta-learner.

`IntermediateFusionType`	Behaviour
`FEATURE_EXTRACTION`	Uses leaf indices or predicted probabilities from base models as features for a meta-learner.
`RESIDUAL`	Each successive modality is trained on the residual of the previous one.
`PROGRESSIVE`	Modalities are processed sequentially and their representations are concatenated progressively.

Hybrid fusion

Hybrid fusion combines early and late strategies in a single configuration.

`HybridFusionType`	Behaviour
`EARLY_LATE`	Some modalities are concatenated (early), others are trained separately (late). A meta-learner stacks all predictions.
`HIERARCHICAL`	Modality groups are fused in a tree structure before a top-level combiner.
`PARALLEL`	All modalities are processed in parallel with independent models. A meta-learner combines the outputs.

FusionConfig parameters


@dataclass
class FusionConfig:
    strategy: FusionStrategy
    fusion_type: Union[EarlyFusionType, LateFusionType, IntermediateFusionType, HybridFusionType]
    model_configs: Optional[dict[str, list[str]]]  # per-modality model overrides
    fusion_params: Optional[dict[str, Any]]          # strategy-specific parameters

The strategy and fusion_type must be compatible (e.g. FusionStrategy.LATE requires a LateFusionType value). A validation error is raised at construction time if they do not match.

Specifying multiple strategies

Pass a list of FusionConfig objects to MultimodalConfig.fusion_configs. The evaluator runs each strategy independently, so results can be compared side by side:


mll.MultimodalConfig(
    modality_configs={...},
    fusion_configs=[
        FusionConfig(strategy=FusionStrategy.EARLY, fusion_type=EarlyFusionType.CONCAT),
        FusionConfig(strategy=FusionStrategy.LATE, fusion_type=LateFusionType.MEAN),
        FusionConfig(strategy=FusionStrategy.LATE, fusion_type=LateFusionType.STACKING),
    ],
)