BBox
We introduce , an any-to-any model that jointly processes a diverse set of 1D, 2D and 3D modalities. can predict any modality conditioned on any one (or multiple) other modalities, enabling cross-modality prediction across dimensionalities. The learned multimodal representations show strong performance on downstream tasks requiring both 2D and 3D understanding, including 3D pose prediction, grasp proposal, and object classification.
We present , an Omni-Dimensional Multimodal Foundation Model capable of any-to-any modelling across a diverse set of 1D, 2D and 3D modalities.
Our method enables any-to-any prediction across:
This any-to-any prediction is enabled by a suite of novel tokenizers we develop for 3D modalities, which enable masked modelling of 3D modalities in the same space as existing 1D and 2D modalities.
Our model is capable of a vast range of tasks out of the box beyond just the standard tasks of 3D generation from images. can perform generation steered by coarse input modalities, decompose objects into parts, retrieve objects & views and much more without any additional training.
Learn how diverse 3D modalities can be tokenized to enable joint modelling across modalities.
See how an encoder-decoder transformer can be adapted to jointly model 1D, 2D and 3D modalities.
Explore the diverse array of tasks an any-to-any 1D-2D-3D model is capable of performing.
Overview
Our model can generate can generate any modality of any dimensionality from any other modality of any dimensionality, 1D, multi-view 2D or 3D. This enables all $n^2$ input-to-output tasks of the $n$ modalities to be performed with a single model.
Hover a row's input (leftmost column) to auto-play every modality
generates from it.
Hover a single cell to inspect one task.
Each cell is a different source to target prediction task, and all cells are generated by a single model. All 2D targets are multiview, which can be seen in the focused display next to the any-to-any grid. Our model is also capable of accepting multiple inputs, and so even this grid is only a subset of the possible tasks it can perform.
To enable multimodal modelling across dimensions, we unify the format of heterogenous modalities by casting them as discrete tokens, which can be jointly modelled with a multimodal masked modelling objective.
can operate across 20 modalities: 3 multi-level and multi-type 1D modalities, 8 multi-view 2D modalities, and 9 3D modalities. These modalities are rendered and pseudo-labelled from meshes, removing the need for a multimodal 3D data source. To train we pseudo-label a 1 million object subset of Objaverse-XL.
By unifying the representation of our data into a shared token space, we can make any-to-any prediction for any input and output tractable regardless of data dimensionality or modality. As tokenizers do not exist for many 3D modalities, we develop our own novel tokenizer architectures for 7 3D modalities. Full details of all the tokenizers used with our model can be found in our paper.
NeRF weights and part features can be represented as triplanes of features, each consisting of 3 dense multi-channel 2D feature tensors. We tokenize these triplanes using a modality-specific ViT-based VQ-VAE tokenizer.
3DGSs can be decoded from a sparse 3D SLAT which encodes high-fidelity 3D details. We densify SLATs via zero padding to obtain a dense 3D grid. Voxels, UDFs and 3D RGB & density grids are cast as dense 3D grids representing occupancy, distance to surface and RGB/density respectively, tokenized with a 3D convolutional VQ-VAE.
2D modalities are obtained from renders and tokenized with pre-existing tokenizers, with finetuning when necessary. We extract camera plucker rays from the rendering pipeline to enable controllable camera viewpoints when modelling 2D data.
The 7 tokenizers introduced by reconstruct their 3D modalities geometrically accurately and to a high level of detail. We open-source the code and model weights for these tokenizers to help enable the easy inclusion of these modalities in other work done by the community
Sample
Voxels, UDFs, 3DGSs and 3D Grids are 3D modalities tokenized as dense 3D, while NeRF weights and Partfield features are 3D modalities tokenized as dense 2D. ULIP Embedding, here represented as its PCA reduced feature vector, is tokenized by an MLP-based tokenizer model.
is a 1 billion parameter transformer encoder-decoder. To perform joint learning on a range of 1D, 2D and 3D modalities, we optimize using the multimodal masked modeling objective introduced in 4M. The tokenized modalities are augmented with a positional embedding corresponding to their structure. We use 2D sin-cos embeddings for 3D modalities cast as dense 2D, 3D sin-cos embeddings for 3D modalities cast as dense 3D and encoded plucker rays for multiview 2D modalities.
As is capable of accepting any combination of modalities both as inputs or outputs, it is capable of performing any prediction or generation task that uses any of the 20 modalities it was trained on. Below we show just a small subset of the possible tasks our model can perform.
Capabilities
can generate anything from either a single view or multiple views from any of the 8 2D modalities it was trained with. This enables the standard Novel View Synthesis (NVS) and 3D generation tasks to be greatly expanded to operate under any modality constraints.
can take a single RGB image as input and lift it into any of the 9 3D modalities our model was trained with. Even though our model was only trained on Objaverse-XL data, it can generalise well to real world data with properties not observed in the training data, such as more varied lighting.
Retrieval preview
Sample
Our model can perform novel view synthesis within the space of any 2D modality, not just RGB like in previous approaches. At any point, these 2D generations can be converted into a 3D model of any kind, which will respect the new geometry and/or colour introduced by the new generated views.
Novel view synthesis can be performed in any modality over arbitrary camera trajectories. When 3D is generated the properties of the novel views will be respected, such as colour and geometry.
Capabilities
As is an any-to-any model, outputs it generates can be passed back to the model to be used as conditioning for generation of other modalities or views of modalities in a process known as chaining.
Chained generation is when generated views or modalities are passed back to the model as additional conditioning for future generation. Here the chain involves the following: 1. The first DINOv2 view is generated conditioned on only the RGB input. 2. The following DINOv2 views are conditioned on the input, this generated DINOv2 view and any other DINOv2 views that have also been generated. 3. The surface normals and RGB views are similarly generated on all modalities and views generated before in addition to the original input. 4. The 3D modalities are generated conditioned on the complete set of input and chained 2D modalities.
Chaining can help improve 's performance on challenging tasks such as 3D generation from a single view input by routing generation though a series of more achievable tasks, rather than forcing zero-shot performance of the task. The benefits of chaining are in general most pronounced when coarser modalities are generated earlier, and details are incrementally built up through modalities. particularly benefits from chaining for generation of complex 3D geometry, where novel view synthesis gives a strong multi-view signal for accurate reconstruction of 3D geometry.
Chaining improves the performance of 3D generation from a single view for challenging inputs. Here each added chaining stage improves the quality of the final 3D model, making it closer to the ground truth model.
Many conditioning modalities and cases are very under-constrained in terms of the objects that could
have given rise to them. These might include caption that only semantically describe an object at a
high level, weak geometry constraints such as 3D bounding boxes and voxels, or input views that leave
aspects of an object ambiguous.
If performing unchained, direct generation our model can utilise this ambiguity to generate a diverse
distribution of other views and modalities that fit the input constraint. Conversely, can perform chained generation to allow each new
generated
output to further constrain the generation conditioning. This leads to consistency across modalities
and views, with one potential sample from the distribution gradually chosen as more constraining
information is added to the conditioning.
Left: Unchained generation predicts each modality and view independently, enabling diversity in outputs. Right: Chained generation constrains outputs to follow all previously generated modalities and views, resulting in consistency between past and future generations.
Capabilities
can also generate anything from any of the 9 3D modalities it was trained with. This enables both a range of new generation tasks, and for many existing tasks involving 3D data to be simplified to a forward pass of our model.
Our model can render any view of any 2D modality from any 3D modality. Even if the 3D modality is untextured, can infer a reasonable texture to apply to the renders.
We can also directly apply textures to any untextured 3D modality instead by generating novel views and textured 3D models jointly conditioned on an untextured 3D modality and a colour palette.
Capabilities
can steer the generation of 2D and 3D modalities from text by jointly conditioning on a weak geometric restriction. This enables objects with specific geometric characteristics to be generated.
Objects generated from more specified restrictions such as voxels or masks strongly follow the geometric constraints, while adapting the texture and content of the object to fit the caption conditioned on.
Objects generated from an even weaker constraint such as conforming to a specific bounding box size along a given axis have freedom to generate with more diversity, while consistently satisfying the constraint.
Left: Objects constrained in the x, y and z axis. Right: The generated fountain takes on different forms when constrained to different extreme axis lengths. To manually control the size constraint, drag a slider.
Capabilities
We can use the multi-level understanding of object parts that our model can produce via it's part feature modality to both split existing 3D models into parts, or generate our own 3D models which can then be segmented to different levels of part granularity.
RGB
Mesh
Part features
Decomposed mesh
Sample
From a single RGB image we can generate both a full object model and its corresponding multi-level part features. These part features can then be clustered to different numbers of parts and applied to the generated model to segment it.
Capabilities
Our model can use any modality as a query to generate DINOv2 and ULIP-2 features, which can be used for retrieval across modalities and dimensions.
ULIP-2 features are global features that align object point cloud features with the CLIP features derived from renders and associated captions of multiple views of the mesh which generated the point cloud. As such, it can be used to query for similar objects across modalities, independent of view.
DINOv2 features can be used to query for similar views across modalities that also contain similar semantic content. Use the arrows to move the query view to explore this matching.
RGB
RGB retrieval (Top-3)
Input modality
Sample
Retrieved modality
Mesh
Mesh retrieval (Top-3)
Input modality
Sample
Retrieved modality
Qualitative Results
generates geometrically consistent 3D outputs from a single view, matching or outperforming existing specialized methods. It especially outperforms other methods when generating 3D models from ambiguous, self-occluded input images.
RGB
Trellis
Sam3D
GPT-6 Astra*
GT
Sample
*Astra performs significantly better with multi-view inputs, while it can struggle with challenging single view inputs such as some of the inputs used here.
Bold values are best in their metric, underlined values are second best. Here chaining involves the generation of five intermediate RGB views around the object.
Quantitative Results
can perform novel view synthesis for any 2D modality. On novel view synthesis of images (RGB), it matches or outperform existing methods on both quality and perceptual consistency metrics.
Bold values are best in their metric, underlined values are second best. Here chaining involves the generation of intermediate DINOv2 features and surface normals for both the input and output views.
We transfer the learned representation of to a range of downstream tasks. This representation proves to be capable of transferring to tasks that require both 2D and 3D understanding, and in particular outperforms on tasks that benefit from effective aggregation of multiview information, such as grasp prediction.
Transfers
Given a RGB image, predict the category-level 6D pose of the 3D objects in the image.
| Method | NOCS MAE ↓ | NOCS PSNR ↑ | Mask IoU (%) ↑ |
|---|
| Method | 3D IoU@25 ↑ | 3D IoU@50 ↑ | 5°5cm ↑ | 10°5cm ↑ | 10°10cm ↑ |
|---|
Open release
We release both a core any-to-any model which is capable of performing all tasks, and a RGB generation specialist which is capable of generating the highest quality RGB outputs.
The core model, capable of any-to-any tasks using all 20 modalities.
View on Hugging Facespecialized to generate high-quality RGB from any input modalities.
View on Hugging FaceWe also release our full suite of novel 3D tokenizers and fine-tuned 2D tokenizers. We hope that this will help further the research of multimodal 3D research by providing an easily adaptable input format for a range of 3D modalities.
3DGS latent tokenizer with an occupancy head, learned with rendering losses.
Link
3DGS latent tokenizer with a flow decoder, for smoothed token interpolation.
Link
Finetuned Surface Normals & Mask 4M tokenizers to adapt to objects.
Link
View on Hugging Face