Aspaklarya — Qwen3-VL-8B for remote-sensing VQA + captioning

LoRA finetunes of Qwen3-VL-8B-Instruct, trained jointly on visual question answering and image captioning over overhead imagery. 17 experiments, 17 evaluated recipes. All model repos are private.

Setup

Winners are merged into the bf16 base and re-validated: adapters are evaluated on a 4-bit base, merged weights are not.

Configurations

Constant everywhere: effective batch 32 (except 014 at 16, 015-017 at 64), warmup 0.03, grad clip 0.3 (1.0 for 011/012), seed 3407.

Results

Best checkpoint per run on a fixed 1000-image draw (2000 graded items), with bootstrap standard error. Overall = mean of caption and VQA. In-dom. = the target overhead-imagery subset (491 images); RSICD = the public RSICD remote-sensing benchmark subset (509 images). Sorted by in-domain overall; bold marks the best value in each column.

Every finetune beats both references overall: +17.1 over the base model and +8.7 over the in-house reference on the full set.

Learning rate at large batch

Aspaklarya-015 gave the best captioning in the table at effective batch 64, but it was trained at 2e-4 — the rate tuned for batch 32. Two follow-ups tested whether the batch-64 recipe was simply under-stepped. Scores below come from a separate 500-image draw used for live checkpoint selection, so they compare only with each other.

Linear LR scaling is wrong for LoRA here: 4e-4 peaked at epoch 1 and then lost ground at every later checkpoint, ending below where 2e-4 started. The adapter's effective step size does not follow the full-finetune scaling rule, so raising LR with batch size overshoots rather than compensating. The intermediate rate 3e-4 traced almost exactly the same curve as 2e-4 and peaked at the same epoch, one tenth of a point lower.

On the shared 1000-image set, neither batch-64 rate beat the reference: 2e-4 is -0.56 overall and -0.09 in-domain, 3e-4 is -0.34 and -0.87. Effective batch 64 is therefore refuted at every learning rate tried, and the loss is consistently in VQA while captioning holds or improves. Together with the batch-16 result this closes the batch-size and learning-rate axes: no hyperparameter lever remains untested, and further gains have to come from the training data.

Error bars

Bootstrap over images, 3000 resamples. Absolute standard error:

Paired model-vs-model error is smaller because image difficulty cancels: 0.42 overall on the full set, 0.56 in-domain. Use the paired figure for any "A beats B" claim.

Caveat

The hyperparameter runs are statistically tied with each other: among runs 001-019, every confidence interval against the leader straddles zero. VQA error is ~2.3x caption error, so no finetune-vs-finetune VQA difference in any column is resolvable; only caption gaps, the gaps to the two references, and the data-scaling result of run 020 are real. Doubling the eval set from 500 to 1000 images did not separate the tied field, and it reshuffled its order.

Findings

• Hyperparameters are saturated; data is not. LR, rank, alpha, dropout, weight decay, gradient clipping, batch geometry, effective batch size, LR scaling at large batch and base-weight precision were all swept without beating the noise floor. Adding 91% more training images cleared it on the first attempt.
• The slices disagree — but data breaks the trade. Ranking by in-domain score almost inverts the RSICD ranking, and every hyperparameter lever that moved one slice moved the other the opposite way. Extra data is the exception: it lifted RSICD by 1.57 while leaving in-domain unchanged.
• Captioning is the real gain. It improves on every slice, by +14 over the base model and +10 to +19 over the in-house reference.
• The in-domain VQA win is not established. The in-house reference posts the single highest in-domain VQA score in the table; it collapses outside that domain, where it falls below the untuned base model.
• The vision tower matters. Freezing it is the one ablation that consistently hurt.
• Extra QA per image does not help. 4-QA and 6-QA mixtures tie with 2-QA.
• Larger batch trades generality for domain fit. Effective batch 64 gives the best in-domain captioning in the table and the worst RSICD VQA.
• Training in bf16 instead of 4-bit is a small, consistent positive — the only change in the whole sweep that never loses on any slice — but it is still inside the error bar.
• Data is the remaining lever, not hyperparameters — and it is not exhausted: the run that used it had not converged when its schedule ended.

Base-weight precision

Runs 001-017 all trained their adapters against 4-bit quantized base weights, while evaluation and the final merge both use the bf16 base. That is a train/serve mismatch: the adapter was fitted to compensate for quantization error that is absent at inference time.

Runs 018 and 019 repeat the two strongest recipes with the base loaded in bf16 and nothing else changed. Against the reference, paired on 1000 images:

These are the first two runs in the sweep to post a positive overall delta, and 018 is the only run that is not worse than the reference on any slice. Both are still well inside the paired error bar (p = 0.34 that each is in fact worse), so neither is promoted.

The mechanism is not the expected one. Removing quantization error should mostly help the harder task, VQA; instead the gain is entirely in captioning (+0.48 and +0.35) while VQA is flat to slightly negative. That is the same caption-up / VQA-flat signature produced by every other lever in the sweep, which suggests precision is acting through the same saturated channel rather than unlocking anything new.

Removing quantization also costs ~9 GB of GPU memory during training (19 GB vs 10 GB), which is a real constraint on a shared 40 GB card.

Training data

Every run up to 017 trained on 6,288 images / 18,864 samples. The source table also carries 5,706 further images that the dataset author left unassigned (Notes == other, RSICD only). They are fully populated, statistically indistinguishable from the assigned training rows, and have zero image-name overlap with either the train or the validation split, so they are safe to train on.

Aspaklarya-020 adds them, taking training data to 11,994 images / 35,981 samples (+91%). Everything else is identical to 018. Paired against the previous selection on the same 1000 images:

This is the first change in the whole programme to clear the error bar. The RSICD interval excludes zero, and the overall caption gain is 3.2 standard errors. It is also the first result that does not trade one slice against the other: in-domain is a clean tie while the general slice moves.

Two further points matter for anyone repeating this:

• The in-domain penalty is transient. At matched optimizer steps the extra data looked actively harmful early — step 2,360 was 1.65 behind on in-domain — and that deficit had vanished by step 4,500. Stopping early would have refuted the best run in the programme.
• The run never converged. Overall score rose at all seven checkpoints, including the last. Four epochs is too short for this data volume, so the ceiling here is unknown.

More data is therefore the live axis, and the only one still producing movement.

Status

The hyperparameter search is closed. Rank, alpha, dropout, learning rate, schedule, weight decay, gradient clipping, vision-tower freezing, QA-per-image, caption prompt pool, batch geometry, effective batch size, learning rate at large batch and base-weight precision were all swept across 19 recipes under identical conditions. None displaced the reference recipe by more than the paired error bar.

Batch size and learning rate illustrate that ceiling. Effective batch 16 came in at -0.59 overall, batch 64 at 2e-4 at -0.56, and batch 64 at 3e-4 at -0.34 overall and -0.87 in-domain; batch 64 at 4e-4 was stopped at epoch 4 while declining. In each case captioning held steady or improved while VQA fell.

Enlarging the training set did what none of those could. The next steps follow from it: train the same corpus for longer, since run 020 had not converged, and add in-domain imagery, since the gain so far is concentrated in the general slice.