sorryhyun/moss-image-colorize

MOSS-Image colorize LoRA (v2)

Colorize v3 is in sorryhyun/MOSS-I2I-LORAs (colorize/), alongside the sanitize and decensor LoRAs and their ComfyUI workflows. This repo keeps v1 and v2.

Two of the twelve held-out pages, seed 1, from v2. Each panel is labelled: the B&W page (the input), the colour target, then v2 with the full colour caption, with a sparse caption, and with the page only.

• Left (touhou). Full caption (layout JSON, see Captions): {"summary":{"series":["touhou"],"caption":"purple eyes, purple hair, purple eyes, black hairband, blue shirt, green hair, yellow shirt, black hat"}}. Sparse: touhou, blue shirt, black hairband.
• Right (blue archive). Full caption: {"summary":{"series":["blue archive"],"caption":"black pantyhose, blue necktie, grey hair, purple eyes, white shirt, white skirt"}}. Sparse: blue necktie, white skirt, purple eyes.

A LoRA for MOSS-Image Base that colorizes a black-and-white manga page or illustration. The page goes in twice: through Qwen3.5's vision tower inside the text encoder, and as a clean VAE latent in the DiT sequence. The output keeps the page's line work, faces and lettering.

Status: research probe. It has been checked by eye on 12 held-out pages, one seed each. Base was not trained for image conditioning. This LoRA teaches it to read a reference.

What changed in v2

	v1	v2
Data	1000 illustration pairs	3334 pairs: 1667 illustrations + 1667 colour manga pages (EN / JA / KO)
Captions	Colour tags	Colour tags in MOSS's layout JSON, plus each line of the page's lettering (OCR) as a text item with its box. 30 % of captioned steps use a tag-dropped variant instead (80 % of tags dropped), with the OCR lines as a text clause
Reference modulation	Ref tokens share the step's timestep	Ref tokens are modulated at t = 0 (a clean image), every block. Image and text keep the step's
Steps	1600, lr 1e-4	10 000 (3 epochs), lr 5e-5

What this changes in use:

• Sparse captions work. Give the few colours you care about (blonde hair, blue bikini). Those are followed, and the rest is filled in plausibly.
• Page only. It now commits to colours. Early in training it fell back to grey and silver.
• Halftone. It turns dot screentone into flat colour.

v2 needs the updated ComfyUI-MOSS-i2i node, the version that reads ref_timestep from the LoRA metadata. An older node runs v2 with the ref at the step's timestep, which is off what it trained on. v1 still loads as before, because it has no ref_timestep in its metadata.

Captions

Besides the page alone, v2 was trained on two kinds of text. Use one of these:

• A few colour tags, plain: touhou, blue shirt, black hairband. The tag-dropped variants were plain tags. On a page with lettering, the variant also carried . OCR result as follows "…" [y0, x0, y1, x1], ….

• The full colour caption as layout JSON. This is how every full caption was trained:

  {"summary":{"series":["touhou"],"caption":"purple hair, blue shirt, green hair, yellow shirt"}}
  

  A page with lettering adds one item per line, boxes in the generated image's pixels with the origin at its centre (yxyx):

  {"summary":{...},"items":[{"label":"text","caption":"text","text":"カタ","bbox":[82,-44,114,-3]}],"bbox_coord_space":"abs_centralized","bbox_format":"yxyx"}
  

A long plain tag list (the first v2's full-caption format) was not a training format for this file and has not been tested with it. Put a long list in the layout JSON. The OCR text is optional: 41 % of training pages had no lettering.

Workflow

Install the custom node from here.

Drag and drop the following image to comfyui.

How it works

MOSS-Image has no image input. Its 4D RoPE (r, f, h, w) has an r axis that its training always set to 0. This LoRA was trained with:

• Text encoder input. The page through the vision tower, then the colour caption, in one user turn: <|im_start|>user\n<|vision_start|>[page]<|vision_end|>{caption}<|im_end|>\n<|im_start|>assistant\n<think>\n. At the target's pixel count, that is ~1000 image tokens at 1 MP. The vision tower alone brings back content but not line placement: one token per 32×32 px.
• VAE reference. The page at half size per side (1/4 area) goes into the DiT sequence as [image | ref | text].
  • The ref tokens get no noise. In v2 they also get their own block modulation from timestep 0, which is the modulation of a clean image.
  • On RoPE they sit at r = 1, with their (h, w) stretched onto the target grid, so each ref token lines up with the region it covers.
  • The ref goes into every CFG branch. The head and the loss read the image tokens only.

Files

File	What
moss_colorize_ref_v2.safetensors	v2. Rank 32, alpha 128, on proj.{q,k,v,o,ffn_up_gate,ffn_down} of all 42 blocks, bf16, 149 MB. Plain lora_unet_layers_* keys; channel scaling is already folded into the stored weights. Metadata ref_timestep = 0.0
moss_colorize_ref_v1.safetensors	v1, same format, shared ref modulation
workflows/moss-i2i-colorize.json	ComfyUI example workflow (loads v2)

ComfyUI

It needs two custom nodes:

• moss-image (OpenMOSS): the MODEL / CLIP / VAE loaders.
• ComfyUI-MOSS-i2i: three nodes for this LoRA.
  • MOSS i2i LoRA Loader. It reads ref_timestep from the metadata.
  • MOSS i2i Text Encode (page + text): the vision-tower encode above.
  • MOSS i2i Reference: the VAE ref, plus an empty latent at the target size.

Wiring (workflows/moss-i2i-colorize.json):

Load Image (page) ──┬─────────────────────────────┐
                    │                             │
MOSS-Image HF Loader ─ MODEL ─ i2i LoRA Loader ─ i2i Reference ─ ModelSamplingSD3(5) ─ SLG(single 22) ─┐
                     ─ CLIP ─┬ i2i Text Encode (page, colour tags) ── positive ─────────────────────────┤
                             └ CLIP Text Encode ("") ──────────────── negative ─────────────────────────┤ KSampler
                     ─ VAE ── i2i Reference ── LATENT ──────────────────────────────────────────────────┘

• Positive. The page plus text in one of the forms under Captions: a few plain colour tags with the copyright first (touhou, purple hair, blue shirt), or the full list as layout JSON. An empty text gives the page-only mode.
• Negative. A plain text encode of "".
• Resolution. Use the same value (1024) on Text Encode and Reference. The page is fit to ~1 MP at its native aspect, with sides multiples of 16.
• Sampler. CFG 3, with block 22 skipped on the uncond branch, and shift 5. The workflow uses ER-SDE at 30 steps, the setting the eval below used. The MOSS Spectrum KSampler (ComfyUI-Spectrum-KSampler) works too.

Training (v2)

Data	3334 synthetic pairs: 1667 illustrations and 1667 colour manga pages (EN / JA / KO). Each colour image is mangafied into a B&W page with dot screentone
Caption	Colour tags: the copyright and comic tags are kept, all others dropped. 30 % of steps use the page with no caption. Of the rest, 70 % use the layout-JSON caption (colour tags + the page's OCR lines with boxes) and 30 % one of 4 tag-dropped variants (80 % of tags dropped) + the OCR lines as a text clause. 59 % of pages have OCR lines
Text encoder	Qwen3.5-9B fp8, GPTQ-rounded (MOSS-Image-Qwen3.5-9B-fp8) for every training encode; vision tower bf16
Steps	10 000 (3 epochs), lr 5e-5, 5 % warmup, max grad norm 1.0
Reference	Half size per side, modulated at timestep 0
Adapter	LoRA r32 / α128, weight-SVD down init, channel scaling α 0.5, DC isolation
σ sampling	logit-normal μ −1.09, s 0.95, cutoff 0.95, 1 % uniform
Cost	8.1 h at 2.91 s/step on one consumer GPU (28 blocks swapped)