Upload folder using huggingface_hub

LICENSE.md

@@ -0,0 +1,35 @@
+# NVIDIA OneWay Noncommercial License
+
+## 1. Definitions
+
+“Licensor” means any person or entity that distributes its Work.
+“Work” means (a) the original work of authorship made available under this license, which may include software, documentation, or other files, and (b) any additions to or derivative works  thereof  that are made available under this license.
+The terms “reproduce,” “reproduction,” “derivative works,” and “distribution” have the meaning as provided under U.S. copyright law; provided, however, that for the purposes of this license, derivative works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work.
+Works are “made available” under this license by including in or with the Work either (a) a copyright notice referencing the applicability of this license to the Work, or (b) a copy of this license.
+
+## 2. License Grant
+
+2.1 Copyright Grant. Subject to the terms and conditions of this license, each Licensor grants to you a perpetual, worldwide, non-exclusive, royalty-free, copyright license to use, reproduce, prepare derivative works of, publicly display, publicly perform, sublicense and distribute its Work and any resulting derivative works in any form.
+
+## 3. Limitations
+
+3.1 Redistribution. You may reproduce or distribute the Work only if (a) you do so under this license, (b) you include a complete copy of this license with your distribution, and (c) you retain without modification any copyright, patent, trademark, or attribution notices that are present in the Work.
+
+3.2 Derivative Works. You may specify that additional or different terms apply to the use, reproduction, and distribution of your derivative works of the Work (“Your Terms”) only if (a) Your Terms provide that the use limitation in Section 3.3 applies to your derivative works, and (b) you identify the specific derivative works that are subject to Your Terms. Notwithstanding Your Terms, this license (including the redistribution requirements in Section 3.1) will continue to apply to the Work itself.
+
+3.3 Use Limitation. The Work and any derivative works thereof only may be used or intended for use non-commercially. Notwithstanding the foregoing, NVIDIA Corporation and its affiliates may use the Work and any derivative works commercially. As used herein, “non-commercially” means for research or evaluation purposes only.
+
+3.4 Patent Claims. If you bring or threaten to bring a patent claim against any Licensor (including any claim, cross-claim or counterclaim in a lawsuit) to enforce any patents that you allege are infringed by any Work, then your rights under this license from such Licensor (including the grant in Section 2.1) will terminate immediately.
+
+3.5 Trademarks. This license does not grant any rights to use any Licensor’s or its affiliates’ names, logos, or trademarks, except as necessary to reproduce the notices described in this license.
+
+3.6 Termination. If you violate any term of this license, then your rights under this license (including the grant in Section 2.1) will terminate immediately.
+
+## 4. Disclaimer of Warranty.
+
+THE WORK IS PROVIDED “AS IS” WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WARRANTIES OR CONDITIONS OF 
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. YOU BEAR THE RISK OF UNDERTAKING ANY ACTIVITIES UNDER THIS LICENSE. 
+
+## 5. Limitation of Liability.
+
+EXCEPT AS PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE SHALL ANY LICENSOR BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF OR RELATED TO THIS LICENSE, THE USE OR INABILITY TO USE THE WORK (INCLUDING BUT NOT LIMITED TO LOSS OF GOODWILL, BUSINESS INTERRUPTION, LOST PROFITS OR DATA, COMPUTER FAILURE OR MALFUNCTION, OR ANY OTHER DAMAGES OR LOSSES), EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

README.md

@@ -1,5 +1,237 @@
 ---
 license: other
-license_name: nscl-v1
-license_link: LICENSE
+license_link: LICENSE.md
+tags:
+  - lighting-estimation
+  - hdr
+  - environment-map
+  - diffusion
+  - pytorch
+  - video
+  - transformer
+  - lora
 ---
+
+# LuxDiT
+
+This is the model checkpoint for [LuxDiT](https://github.com/nv-tlabs/LuxDiT): Lighting Estimation with Video Diffusion Transformer. It is finetuned on image data and includes a LoRA adapter for real scenes.
+
+![](https://research.nvidia.com/labs/toronto-ai/LuxDiT/assets/teaser.png)
+
+## Model description
+
+LuxDiT is a generative lighting estimation model that predicts high-quality HDR environment maps from visual input. It produces accurate lighting while preserving scene semantics, enabling realistic virtual object insertion under diverse lighting conditions. This model is ready for non-commercial use.
+
+- **Checkpoint**: Video model (image and video-finetuned)
+- **LoRA**: Included for real-scene generalization
+- **Paper**: [LuxDiT: Lighting Estimation with Video Diffusion Transformer](https://arxiv.org/abs/2509.03680)
+- **Project page**: https://research.nvidia.com/labs/toronto-ai/LuxDiT/
+
+**Use case:** LuxDiT supports studies and prototyping in video lighting estimation. This release is an open-source implementation of our research paper, intended for AI research, development, and benchmarking for lighting estimation research.
+
+### Model architecture
+
+- **Architecture type:** Transformer (based on [CogVideoX](https://github.com/THUDM/CogVideoX))
+- **Parameters:** 5B
+- **Input:** RGB video frames; shape `[batch_size, num_frames, height, width, 3]`; recommended resolution 480×720
+- **Output:** RGB video frames (dual tonemapped LDR and log); output resolution 256×512; use the HDR merger to obtain `.exr` HDR envmaps
+
+### Software and hardware
+
+- **Runtime:** Python and PyTorch
+- **Supported hardware:** NVIDIA Ampere (e.g. A100 GPUs)
+- **Operating system:** Linux
+
+## How to use
+
+### Download from Hugging Face
+
+From the [LuxDiT repository](https://github.com/nv-tlabs/LuxDiT) root:
+
+```bash
+python download_weights.py --repo_id <HF_ORG>/LuxDiT
+```
+
+This saves the checkpoint to `checkpoints/LuxDiT` by default. Use `--local_dir` to override.
+
+### Image Inference: synthetic images (in-domain)
+
+```bash
+DIT_PATH=checkpoints/LuxDiT/luxdit_image
+INPUT_DIR=examples/input_demo/synthetic_images
+OUTPUT_DIR=test_output/synthetic_images
+
+python inference_luxdit.py \
+    --config configs/luxdit_base.yaml \
+    --transformer_path $DIT_PATH \
+    --input_dir $INPUT_DIR \
+    --output_dir $OUTPUT_DIR \
+    --resolution 480 720 \
+    --guidance_scale 2.5 \
+    --num_inference_steps 50 \
+    --seed 33
+
+python hdr_merger.py \
+    --model_path checkpoints/hdr_merge_mlp \
+    --input_dir $OUTPUT_DIR/ldr_log \
+    --output_dir $OUTPUT_DIR/hdr
+```
+
+### Image Inference: real scenes (with LoRA)
+
+Use the LoRA adapter in this checkpoint for better generalization to real photos:
+
+```bash
+DIT_PATH=checkpoints/LuxDiT/luxdit_image
+LORA_PATH=checkpoints/luxdit_image/lora
+INPUT_DIR=examples/input_demo/scene_images
+OUTPUT_DIR=test_output/scene_images
+
+python inference_luxdit.py \
+    --config configs/luxdit_base.yaml \
+    --transformer_path $DIT_PATH \
+    --lora_dir $LORA_PATH \
+    --lora_scale 0.8 \
+    --input_dir $INPUT_DIR \
+    --output_dir $OUTPUT_DIR \
+    --resolution 480 720 \
+    --guidance_scale 2.5 \
+    --num_inference_steps 50 \
+    --seed 33
+
+python hdr_merger.py \
+    --input_dir $OUTPUT_DIR/ldr_log \
+    --output_dir $OUTPUT_DIR/hdr
+```
+
+Adjust `lora_scale` (e.g. 0.0–1.0) to control how much the input scene is merged into the estimated envmap.
+
+
+### Video Inference: synthetic videos (in-domain)
+
+Requires `--data_type video`:
+
+```bash
+DIT_PATH=checkpoints/LuxDiT/luxdit_video
+INPUT_DIR=examples/input_demo/synthetic_videos
+OUTPUT_DIR=test_output/synthetic_videos
+
+python inference_luxdit.py \
+    --config configs/luxdit_base.yaml \
+    --transformer_path $DIT_PATH \
+    --input_dir $INPUT_DIR \
+    --output_dir $OUTPUT_DIR \
+    --resolution 480 720 \
+    --guidance_scale 2.5 \
+    --num_inference_steps 40 \
+    --seed 33 \
+    --data_type video
+
+python hdr_merger.py \
+    --input_dir $OUTPUT_DIR/ldr_log \
+    --output_dir $OUTPUT_DIR/hdr
+```
+
+### Video Inference: real scenes from video (with LoRA)
+
+Use the LoRA adapter in this checkpoint for better generalization to real video:
+
+```bash
+DIT_PATH=checkpoints/LuxDiT/luxdit_video
+LORA_PATH=checkpoints/LuxDiT/luxdit_video/lora
+INPUT_DIR=examples/input_demo/scene_videos
+OUTPUT_DIR=test_output/scene_videos
+
+python inference_luxdit.py \
+    --config configs/luxdit_base.yaml \
+    --transformer_path $DIT_PATH \
+    --lora_dir $LORA_PATH \
+    --lora_scale 0.8 \
+    --input_dir $INPUT_DIR \
+    --output_dir $OUTPUT_DIR \
+    --resolution 480 720 \
+    --guidance_scale 2.5 \
+    --num_inference_steps 40 \
+    --seed 33 \
+    --data_type video
+
+python hdr_merger.py \
+    --input_dir $OUTPUT_DIR/ldr_log \
+    --output_dir $OUTPUT_DIR/hdr
+```
+
+Adjust `lora_scale` (e.g. 0.0–1.0) to control how much the input scene is merged into the estimated envmap.
+
+### Inference: object video scan with camera poses
+
+For multi-view object captures, you can optionally provide camera poses (`--camera_pose_file`, OpenCV format) to align the estimated envmap to a canonical layout:
+
+```bash
+DIT_PATH=checkpoints/LuxDiT/luxdit_video
+LORA_PATH=checkpoints/LuxDiT/luxdit_video/lora
+INPUT_DIR=examples/input_demo/object_scans/antman
+OUTPUT_DIR=test_output/object_scans/antman
+CAM_FILE=examples/input_demo/object_scans/antman/antman.camera.json
+
+python inference_luxdit.py \
+    --config configs/luxdit_base.yaml \
+    --transformer_path $DIT_PATH \
+    --lora_dir $LORA_PATH \
+    --lora_scale 0.0 \
+    --input_dir $INPUT_DIR \
+    --output_dir $OUTPUT_DIR \
+    --resolution 512 512 \
+    --guidance_scale 2.5 \
+    --num_inference_steps 40 \
+    --seed 33 \
+    --data_type video \
+    --camera_pose_file $CAM_FILE
+
+python hdr_merger.py \
+    --input_dir $OUTPUT_DIR/ldr_log \
+    --output_dir $OUTPUT_DIR/hdr
+```
+
+## Related checkpoints
+
+| Checkpoint   | Description |
+|-------------|-------------|
+| [luxdit_image](https://huggingface.co/nvidia/LuxDiT/luxdit_image) | Image-finetuned, with LoRA for real scenes |
+| [luxdit_video](https://huggingface.co/nvidia/LuxDiT/luxdit_video) | Video-finetuned, with LoRA for real scenes |
+
+For video inputs and object scans, use **luxdit_video** instead.
+
+## Training data
+
+This checkpoint is video-finetuned on the **SyntheticScenes** dataset:
+
+- **Modality:** Video
+- **Scale:** ~108,000 rendered videos; each video has 57 frames at 704×1280 resolution
+- **Collection:** Synthetic data generated with an OptiX-based physically based path tracer
+- **Labels:** Produced by the renderer (no manual labeling)
+- **Per sample:** Input RGB (LDR) video and HDR environment lighting
+
+Testing and evaluation use held-out 10% splits of the same dataset.
+
+## Output format
+
+The model outputs dual tonemapped environment maps (LDR and log); use the HDR merger to get `.exr` HDR envmaps. By default, the camera pose of the input image defines the world frame. See the [main README](https://github.com/NVIDIA/LuxDiT#output-format) for the exact layout of LDR/log vs merged HDR.
+
+## Ethical considerations
+
+NVIDIA believes Trustworthy AI is a shared responsibility. When using this model in accordance with the terms of service, ensure it meets requirements for your use case and addresses potential misuse. You are responsible for having proper rights and permissions for all input image and video content; if content includes people, personal health information, or intellectual property, generated outputs will not blur or preserve proportions of subjects. Users are responsible for model inputs and outputs and for implementing appropriate guardrails and safety mechanisms before deployment. To report model quality, risk, security vulnerabilities, or other concerns, see [NVIDIA AI Concerns](https://app.intigriti.com/programs/nvidia/nvidiavdp/detail).
+
+## License
+
+NVIDIA OneWay Noncommercial License. See the [LICENSE](LICENSE.md) in the LuxDiT repository.
+
+## Citation
+
+```bibtex
+@article{liang2025luxdit,
+  title={Luxdit: Lighting estimation with video diffusion transformer},
+  author={Liang, Ruofan and He, Kai and Gojcic, Zan and Gilitschenski, Igor and Fidler, Sanja and Vijaykumar, Nandita and Wang, Zian},
+  journal={arXiv preprint arXiv:2509.03680},
+  year={2025}
+}
+```

luxdit_image/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

luxdit_image/config.json

@@ -0,0 +1 @@
+{"_class_name": "CustomCogVideoXTransformer3DModel", "_diffusers_version": "0.32.1", "_name_or_path": "THUDM/CogVideoX-5b-I2V", "activation_fn": "gelu-approximate", "additional_output_channels": 32, "additional_patch_embed_channels": 48, "attention_bias": true, "attention_head_dim": 64, "dropout": 0.0, "flip_sin_to_cos": true, "freq_shift": 0, "in_channels": 16, "max_text_seq_length": 226, "norm_elementwise_affine": true, "norm_eps": 1e-05, "num_attention_heads": 48, "num_layers": 42, "ofs_embed_dim": null, "out_channels": 16, "patch_bias": true, "patch_size": 2, "patch_size_t": null, "sample_frames": 49, "sample_height": 60, "sample_width": 90, "spatial_interpolation_scale": 1.875, "temporal_compression_ratio": 4, "temporal_interpolation_scale": 1.0, "text_embed_dim": 4096, "time_embed_dim": 512, "timestep_activation_fn": "silu", "use_fixed_pos_embedding": false, "use_learned_positional_embeddings": false, "use_positional_embeddings": false, "use_rotary_positional_embeddings": true}

luxdit_image/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:14373cebcb5b05880eba20b5015cd938adb65d9871131bfedcca237a724cceea
+size 11148972264

luxdit_image/lora/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ec8410c0f6bc11f070c7f516e49ebfdd28d6a3dd3a39959abb5ed25747a2f7bb
+size 594609352

luxdit_video/.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

luxdit_video/config.json

@@ -0,0 +1 @@
+{"_class_name": "CustomCogVideoXTransformer3DModel", "_diffusers_version": "0.32.1", "_name_or_path": "THUDM/CogVideoX-5b-I2V", "activation_fn": "gelu-approximate", "additional_output_channels": 32, "additional_patch_embed_channels": 48, "attention_bias": true, "attention_head_dim": 64, "dropout": 0.0, "flip_sin_to_cos": true, "freq_shift": 0, "in_channels": 16, "max_text_seq_length": 226, "norm_elementwise_affine": true, "norm_eps": 1e-05, "num_attention_heads": 48, "num_layers": 42, "ofs_embed_dim": null, "out_channels": 16, "patch_bias": true, "patch_size": 2, "patch_size_t": null, "sample_frames": 49, "sample_height": 60, "sample_width": 90, "spatial_interpolation_scale": 1.875, "temporal_compression_ratio": 4, "temporal_interpolation_scale": 1.0, "text_embed_dim": 4096, "time_embed_dim": 512, "timestep_activation_fn": "silu", "use_fixed_pos_embedding": false, "use_learned_positional_embeddings": false, "use_positional_embeddings": false, "use_rotary_positional_embeddings": true}

luxdit_video/diffusion_pytorch_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:fb1fe4b81b0d2f70c55088093553ea826adba06203168295a49d8ae54dae5a08
+size 11148972264

luxdit_video/lora/model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:bed86768b8b6b285697217f860baa5658ce81e41082fb17df6c6009dbbcb8d08
+size 594609352