BBox
We introduce , an any-to-any model that jointly processes a diverse set of 1D, 2D and 3D modalities. can predict any modality conditioned on any one (or multiple) other modalities, enabling cross-modality prediction across dimensionalities. The learned multimodal representations show strong performance on downstream tasks requiring both 2D and 3D understanding, including 3D pose prediction, grasp proposal, and object classification.
We present , an Omni-Dimensional Multimodal Foundation Model capable of any-to-any modelling across a diverse set of 1D, 2D and 3D modalities.
Our method enables any-to-any prediction across:
This any-to-any prediction is enabled by a suite of novel tokenizers we delevop for 3D modalities, which enable masked modeling of 3D modalities in the same space as existing 1D and 2D modalities.
Our model is capable of a vast range of tasks out of the box beyond just the standard tasks of 3D generation from images. can perform generation steered by coarse input modalities, decompose objects into parts, retrieve views and objects and much more without any additional training.
Learn how diverse 3D modalities can be tokenized to enable joint modelling across modalities.
See how an encoder-decoder transformer can be adapted to jointly model 1D, 2D and 3D modalities.
Explore the diverse array of tasks an any-to-any 1D-2D-3D model is capable of performing.
Overview
Our model can generate can generate any modality of any dimensionality from any other modality of any dimensionality, 1D, multi-view 2D or 3D. This enables all $n^2$ input-to-output tasks of the $n$ modalities to be performed with a single model.
Hover a row's input (leftmost column) to auto-play every modality
generates from it.
Hover a single cell to inspect one task.
Each cell is a different source to target prediction task, and all cells are generated by a single model. All 2D targets are multiview, which can be seen in the focused display next to the any-to-any grid. Our model is also capable of accepting multiple inputs, and so even this grid is only a subset of the possible tasks it can perform.
To enable multimodal modelling across dimensions, we unify the format of heterogenous modalities by casting them as discrete tokens, which can be jointly modelled with a multimodal masked modelling objective.
can operate across 20 modalities, 3 multi-level and multi-type 1D modalities, 8 multi-view 2D modalities, and 9 3D modalities. These modalities are rendered and pseudo-labelled from meshes, removing the need for a multimodal 3D data source.
By unifying the representation of our data into a shared token space, we can make any-to-any prediction for any input and output tractable regardless of data dimensionality or modality. As tokenizers do not exist for many 3D modalities, we develop our own novel tokenizer architectures for 7 3D modalities.
NeRF weights and part features can be represented as triplanes of features, each consisting of 3 dense multi-channel 2D feature tensors. We tokenize these triplanes using a modality-specific ViT-based VQ-VAE tokenizer.
3DGSs can be decoded from a sparse 3D SLAT which encodes high-fidelity 3D details. We densify SLATs via zero padding to obtain a dense 3D grid. Voxels, UDFs and NeRF grids are cast as dense 3D grids representing occupancy, distance to surface and RGB/density respectively, tokenized with a 3D convolutional VQ-VAE.
2D modalities are obtained from renders and tokenized with pre-existing tokenizers, with finetuning when necessary. We extract camera plucker rays from the rendering pipeline to help with modelling.
The 7 tokenizers introduced by reconstruct their 3D modalities geometrically accurately and to a high level of detail.
Sample
Voxels, UDFs, 3DGSs and 3D Grids are 3D modalities tokenized as dense 3D, while NeRF weights and Partfield features are 3D modalities tokenized as dense 2D. ULIP Embedding, here represented as its PCA reduced feature vector, is tokenized by an MLP-based tokenizer model.
is a 1 billion parameter transformer encoder-decoder. To perform joint learning on a range of 1D, 2D and 3D modalities, we optimize with the multimodal masked modeling objective introduced in 4M. The tokenized modalities are augmented with a positional embedding corresponding to their structure. We use 2D sin-cos embeddings for 3D modalities cast as dense 2D, 3D sin-cos embeddings for 3D modalities cast as dense 3D and encoded plucker rays are used for multiview 2D modalities.
As is capable of accepting any combination of modalities both as inputs or outputs, it is capable of performing any prediction or generation task that uses any of the 20 modalities it was trained on. Below we show some of the possible tasks our model can do, but these are only a very small subset of our model’s capabilities.
Capabilities
can generate anything from any of the 8 2D modalities it was trained with. This enables the standard Novel View Synthesis (NVS) and 3D generation tasks to be greatly expanded to operate under any modality constraints.
can take a single image and lift it into any of the 9 3D modalities our model was trained with. This image input can also be a real world image when paired with an approach to segment out the target object.
Retrieval preview
Our model can perform novel view synthesis within the space of any 2D modality, or any other 2D modality if an input is first translated to that modality. At any point, these 2D generations can be converted into a 3D model of any kind, which will respect the constraints introduced by the novel view synthesis generations.
RGB image inputs can be translated into other 2D modalities by our model. Novel view synthesis can then be performed in the new modality over arbitrary camera trajectories. When 3D is generated the properties of the novel views will be respected, such as colour and geometry.
Capabilities
As is an any-to-any model, outputs it generates can be passed back to the model to be used as conditioning for generation of other modalities or views of modalities in a process known as chaining.
Chained generation is when generated views or modalities are passed back to the model as additional conditioning for future generation. Here the chain involves the following: 1. The first DINOv2 view is generated conditioned on only the RGB input. 2. The following DINOv2 views are conditioned on the input, this generated DINOv2 view and any other DINOv2 views that have also been generated. 3. The surface normals and RGB views are similarly generated on all modalities and views generated before in addition to the original input. 4. The 3D modalities are generated conditioned on the complete set of input and chained 2D modalities.
Chaining can help improve 's performance on challenging tasks such as 3D generation from a single view input by routing generation though a series of more achievable tasks, rather than forcing zero-shot performance of the task. The benefits of chaining are in general most pronounced when coarser modalities are generated earlier, and details are incrementally built up through modalities. particularly benefits from chaining for generation of complex 3D geometry, where novel view synthesis gives a strong multi-view signal for accurate reconstruction of 3D geometry.
Chaining improves the performance of 3D generation from a single view for challenging inputs. Here each added chaining stage improves the quality of the final 3D model, making it closer to the ground truth model.
Many conditioning modalities and cases are very under-constrained in terms of the objects that could
have given rise to them. These might include caption that only semantically describe an object at a
high level, weak geometry constraints such as 3D bounding boxes and voxels, or input views that leave
aspects of an object ambiguous.
If performing unchained, direct generation our model can utilise this ambiguity to generate a diverse
distribution of other views and modalities that fit the input constraint. Conversely, can perform chained generation to allow each new generated
output to further constrain the generation conditioning. This leads to consistency across modalities
and views, with one potential sample from the distribution gradually chosen as more constraining
information is added to the conditioning.
Left: Unchained generation predicts each modality and view independently, enabling diversity in outputs. Right: Chained generation constrains outputs to follow all previously generated modalities and views, resulting in consistency between past and future generations.
Capabilities
can also generate anything from any of the 9 3D modalities it was trained with. This enables both a range of new generation tasks and for many existing tasks involving 3D data to be simplified to a single forward pass of our model.
Our model can render any view of any 2D modality from any 3D modality. Even if the 3D modality is untextured, can infer a reasonable texture to apply to the renders.
We can also directly apply textures to 3D models instead by generating novel views and textured 3D models jointly conditioned on an untextured 3D modality and a colour palette.
Capabilities
can steer the generation of 2D and 3D modalities from text by jointly conditioning on a weak geometric restriction. This enables objects with specific geometric characteristics to be generated.
Objects generated from more specified restrictions such as voxels or masks strongly follow the geometric constraints, while adapting the texture and content of the object to fit the caption conditioned on.
Objects generated from a weaker constraint such as conforming to a specific bounding box size along a given axis have freedom to generate with more diversity, while consistently satisfying the constraint.
Left: Objects constrained in the x, y and z axis. Right: The generated fountain takes on different forms when constrained to different extreme axis lengths.
Capabilities
We can use the multi-level understanding of object parts that our model can produce via it’s part feature modality to both split existing 3D models into parts, or generate our own 3D models which can then be segmented to different levels of part granularity.
RGB
Mesh
Part features
Decomposed mesh
Sample
From a single RGB image we can generate both a full object model and its corresponding multi-level part features. These part features can then be clustered to different numbers of parts and applied to the generated model to segment it.
Capabilities
Our model can use any modality as a query to generate DINOv2 and ULIP-2 features, which can be used for retrieval across modalities and dimensions.
ULIP-2 features are global features that align object point cloud features with the CLIP features derived from renders and associated captions of multiple views of the mesh which generated the point cloud. As such, it can be used to query for similar objects across modalities, independent of view.
DINOv2 features can be used to query for similar views across modalities that also contain similar semantic content. Use the arrows to move the query view to explore this matching.
RGB
RGB retrieval (Top-3)
Input modality
Sample
Retrieved modality
Mesh
Mesh retrieval (Top-3)
Input modality
Sample
Retrieved modality
Qualitative Results
generates geometrically consistent 3D outputs from a single view, matching or outperforming existing specialized methods.
RGB
SAM3D
TRELLIS
Ours
GT
Sample
Quantitative Results
For single-view 3D generation, we compare the quality and consistency of reconstructed outputs against established specialized methods.
Bold, shaded cells are best in their metric; underlined values are second-best. Rows marked (Chained) use chained generation.
Quantitative Results
Starting from a single input view, we evaluate the quality and perceptual consistency of images rendered from unseen viewpoints.
Bold, shaded cells are best in their metric; underlined values are second-best. Rows marked (Chained) use chained generation.
transfers multimodal representations to downstream tasks and acts as a strong representation capable of transferring to tasks that require both 2D and 3D understanding.
Transfers
Given a RGB image, predict the category-level 6D pose of the 3D objects in the image.
| Method | NOCS MAE ↓ | NOCS PSNR ↑ | Mask IoU (%) ↑ |
|---|
| Method | 3D IoU@25 ↑ | 3D IoU@50 ↑ | 5°5cm ↑ | 10°5cm ↑ | 10°10cm ↑ |
|---|
Open release
We release both a core any-to-any model which is capable of performing all tasks, and a RGB generation specialist which is capable of generating the highest quality RGB outputs.
The core model, capable of any-to-any tasks using all 20 modalities.
View on Hugging Facespecialized to generate high-quality RGB from any input modalities.
View on Hugging FaceWe also release our full suite of novel 3D tokenizers and fine-tuned 2D tokenizers. We hope that this will help further the research of multimodal 3D research by providing an easily adaptable input format for a range of 3D modalities.
3DGS latent tokenizer with an occupancy head, learned with rendering losses.
Link
3DGS latent tokenizer with a flow decoder, for smoothed token interpolation.
Link
Finetuned Surface Normals & Mask 4M tokenizers to adapt to objects.
Link
View on Hugging Face