MaPa

Preview：

<br />

Introduce：

MaPa is an innovative way to generate materials for 3D meshes based on text descriptions. This technique represents appearance by creating segmented, programmatic material maps, supports high-quality rendering, and provides significant flexibility in editing. Using a pre-trained 2D diffusion model, MaPa Bridges the gap between text descriptions and material maps without the need for large amounts of paired data. The technique synthesizes a 2D image aligned with the grid portion by decomposing the shape into parts and designing a diffusion model for the control segment, then initializes the parameters of the material map and fine-tunes it through a differentiable rendering module to produce materials that match the text description. Extensive experiments have shown that MaPa is superior to prior art in terms of fidelity, resolution, and editability.
MaPa
https://pic.chinaz.com/ai/2024/06/24061109225164623246.jpg
Stakeholders:
MaPa is for 3D designers and visual effects experts who need to quickly and flexibly create realistic 3D materials based on text descriptions. This technology simplifies the traditional material production process and increases work efficiency while maintaining the high quality and customizability of the material.
Usage Scenario Examples:

Designers use MaPa to create realistic 3D character skin textures for upcoming movies.
The visual effects studio uses MaPa to quickly generate scene textures that match the description of the script.
Game developers use MaPa to add high-resolution and editable materials to virtual environments in games.

The features of the tool:

Text-driven material generation: Automatically generates 3D shaped materials based on text descriptions.
Segmented controlled image generation: Use ControlNet technology to break down the 3D mesh into different parts and project it onto the 2D image.
Classification and optimization of material properties and appearance: Groups parts of the grid with similar properties, and selects and optimizes material maps to accurately represent their textures and properties.
Iterative rendering and refinement: Continuously render and refine the material map through multiple views, filling in gaps in visual data, and repeating the grouping and optimization phases.
Gpt-4-based object editing: text-driven material editing via predefined apis and GPT-4.
Differentiable rendering module: Used to fine-tune the material map parameters to ensure that the material conforms to the text description.

Steps for Use:

Visit the MaPa website and register an account.
Upload or create a 3D mesh model.
Enter a text description that describes the material.
Create a preliminary material map using MaPa’s auto-generation feature.
Edit and adjust the generated materials as needed.
Fine-tune the material map with differentiable rendering module to ensure that the material matches the text description.
After editing the material, export the material and apply it to the 3D model.

Tool’s Tabs: 3D design, text to material