Hugging Face's logo

Apryle
/
AVTime

video-understanding
multimodal
temporal-reasoning
qwen
Model card Files
xet
 Community

You need to agree to share your contact information to access this model

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Log in or Sign Up to review the conditions and access this model content.

AVTime: data construction, pure BiTSC training, and AVTime-Bench

This repository is the code-only release of the AVTime project. It covers the complete engineering path used by the project:

  1. build the AVTime-50K train/test split;
  2. cut complete videos and produce segment-level captions;
  3. train the actor with SFT and pure BiTSC GRPO;
  4. run LongVALE-compatible dense video captioning inference;
  5. run the canonical AVTime-Bench inference, Qwen3.6-27B judging, and scoring pipeline.

The caption pipeline explicitly includes both clipping and segmented captioning. Clipping is performed by AVCut, and every rendered segment is then captioned before its local timestamps are mapped back to the complete video timeline.

Release status. Hugging Face is public with manual gated access. The matching GitHub mirror is private while the paper is under review and is intended to become public at release time.

Code-only boundary. Videos, full annotations, model checkpoints, generated captions, benchmark predictions, judge responses, caches, and credentials are not included. The bundled files contain source code, configuration, tests, and the frozen 300-video split ID list only.

For a component-by-component explanation of why the system is structured this way, read the Chinese design document. Shorter component-specific notes remain in data/README.md, data/caption_pipeline/README.md, training/README.md, and benchmark/README.md.

Contents

System overview 

complete videos
    |
    +-- AVTime-50K split --------------------------------------+
    |       train.jsonl / test.jsonl / manifest                |
    |                                                          |
    +-- AVCut                                                  |
    |       ASR pauses + scene changes + audio events           |
    |       -> clips/*.mp4 + clips/*.mp3 + segments.json        |
    |                                                          |
    +-- segmented captioning                                   |
            Qwen3-Omni or Gemini                               |
            -> captions_merged.json on the full-video timeline |
                                                               |
annotations + media references                                 |
    |                                                          |
    +-- one-round SFT data -> SFT actor                         |
    |                                                          |
    +-- pure BiTSC data -> GRPO + avtime_btsc ORM               |
                                                               |
trained actor -------------------------------------------------+
    |
    +-- LongVALE-compatible DVC inference + official merge
    |
    +-- AVTime-Bench HQ300
            target inference
            -> official LongVALE adjacent-caption merge
            -> Qwen3.6-27B caption-only judge
            -> deterministic T2E/E2T/E2E scoring

The three stages exchange explicit artifacts rather than hidden process state:

Stage Main input Main output Contract
Split complete-video JSONL train.jsonl, test.jsonl video-disjoint rows and recorded hashes
Clipping one complete video segments.json and rendered clips full coverage and non-overlapping segments
Captioning segmented video directory captions_merged.json seconds on the original video timeline
SFT build AVTime/LongVALE rows sft_train.jsonl one-round From XX to YY answers
GRPO build SFT rows bitsc_grpo.jsonl schema avtime_btsc_grpo_v1, task bitsc only
Inference reference-free video rows raw model answers no assistant/reference leakage
Normalization raw answers canonical event JSONL frozen percentage-time conversion and merge
Evaluation predictions + private references six scores and traces judge handles semantics; code handles time

Repository layout 

.
├── data/
│   ├── split_avtime_50k.py
│   ├── splits/
│   │   └── avtime_bench_hq300_zeroshot_ids.jsonl
│   └── caption_pipeline/
│       ├── avcut_config.yaml
│       ├── requirements*.txt
│       └── src/pipeline/
│           ├── avcut.py
│           ├── avcut_batch.py
│           ├── cut.sh
│           ├── cut_multinode.sh
│           ├── qwen3_caption.py
│           ├── qwen3_caption_multinode.sh
│           ├── qwen3_caption_progress.py
│           └── gemini_caption.py
├── training/
│   ├── sft/
│   │   ├── build_sft_dataset.py
│   │   └── train_sft.sh
│   └── bitsc/
│       ├── build_grpo_dataset.py
│       ├── bitsc_orm.py
│       ├── serve_rollout.sh
│       ├── serve_reward_model.sh
│       └── train_grpo.sh
├── benchmark/
│   ├── build_target_inputs.py
│   ├── infer_qwen3_omni.py
│   ├── normalize_predictions.py
│   ├── judge_qwen36.py
│   ├── score_avtime_bench.py
│   ├── run_avtime_bench.sh
│   ├── run_avtime_bench_structured.sh
│   └── prompts/
├── tests/
├── docs/DESIGN_ZH.md
├── PROVENANCE.md
└── RELEASE_CHECKLIST.md

Run every command below from the repository root unless a command says otherwise.

Installation

1. System requirements

The CPU-only builders and tests require Python 3.10 or newer. Video processing also requires ffmpeg and ffprobe on PATH:

ffmpeg -version
ffprobe -version
python3 --version

The clipping, Qwen3-Omni inference, SFT, and GRPO stages are GPU workloads. Install a CUDA-compatible PyTorch build for the target machine before installing the remaining GPU packages.

2. Lightweight package

For JSONL builders, normalization, scoring, and unit tests:

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade pip
python3 -m pip install -e .
python3 -m pip install -r requirements-dev.txt

The lightweight project dependencies are intentionally limited to PyYAML, HTTPX, and the OpenAI Python client. They do not install a universal GPU stack.

3. Caption pipeline

Install the caption client:

python3 -m pip install -r data/caption_pipeline/requirements.txt

Install AVCut after installing the correct PyTorch and TorchAudio wheels:

python3 -m pip install -r data/caption_pipeline/requirements-avcut.txt

The default AVCut configuration uses NVIDIA Parakeet through NeMo. WhisperX is an optional ASR backend and is not installed by the provided requirements file. PySceneDetect provides scene cuts, while PANNs provides music and sound effect transitions.

PANNs does not download its checkpoint in this release. Provide either:

export PANNS_CHECKPOINT_PATH=/absolute/path/to/Cnn14_mAP=0.431.pth

or place that checkpoint under:

${AVCUT_PANNS_HOME}/Cnn14_mAP=0.431.pth

If AVCUT_PANNS_HOME is unset, use the location expected by your local configuration.

4. Training and benchmark GPU stacks

The launchers expect compatible installations of:

  • PyTorch and the CUDA runtime for the machine;
  • Megatron-SWIFT for SFT and GRPO;
  • vLLM for rollout, reward-model service, target inference, and judge inference;
  • Transformers with Qwen3-Omni support;
  • qwen_omni_utils for multimodal preprocessing.

These packages evolve quickly and have checkpoint- and hardware-specific compatibility constraints. The repository therefore does not claim that a single lockfile is valid across all supported clusters. Record the exact environment used for every reported run.

Required private assets

Create or point the commands at equivalent locations. The following layout shows the default paths:

data/
├── AVTime-50K/
│   └── all.jsonl
├── LongVALE/
│   └── train_agent.jsonl
└── AVTime-Bench/
    ├── test.jsonl
    ├── inventory.jsonl
    ├── benchmark_reference_manifest.jsonl
    └── target_inputs.jsonl              # generated when absent

/absolute/path/to/source_videos/
/absolute/path/to/Qwen3-Omni-checkpoint/
/absolute/path/to/SFT-actor/
/absolute/path/to/reward-model/

The source media paths may live anywhere. Benchmark inventory rows must point to existing absolute paths. Do not commit private media, annotations, credentials, or generated judge artifacts.

Part I: data construction

1. AVTime-50K split

What the splitter does

data/split_avtime_50k.py reads one complete-video record per JSONL line. It selects AVTime rows conservatively:

  1. prefer metadata.source == "avtime";
  2. when source metadata has been stripped, require the public avtime_ ID namespace;
  3. exclude unknown unlabeled rows instead of guessing their source.

Each retained row must provide:

  • a non-empty unique ID;
  • one resolvable input-video reference (from metadata.input_video, otherwise the first entry in videos);
  • positive duration metadata;
  • a positive event count, either in metadata.event_count or recoverable by scanning assistant messages from the end for the latest parseable, non-empty event JSON.

The default run asserts a 49,977-row AVTime corpus. It uses the bundled frozen 300-ID zero-shot manifest and produces 49,677 training rows plus 300 benchmark rows.

Reproduce the frozen public split

Place the complete annotation file at data/AVTime-50K/all.jsonl, then run:

python3 -m data.split_avtime_50k

Equivalent explicit command:

python3 -m data.split_avtime_50k \
  --source data/AVTime-50K/all.jsonl \
  --output-root outputs/avtime-50k-split \
  --test-ids data/splits/avtime_bench_hq300_zeroshot_ids.jsonl \
  --expected-rows 49977

The output directory contains:

outputs/avtime-50k-split/
├── train.jsonl
├── test.jsonl
├── test_ids.jsonl
└── split_manifest.json
  • train.jsonl and test.jsonl preserve the complete source rows.
  • test_ids.jsonl contains the ID, video reference, duration, event count, and the assigned duration/event-count quantile bins.
  • split_manifest.json records schema version, source and output SHA-256 values, split sizes, distributions, and disjointness checks.

Derive a new deterministic holdout

This is useful for an ablation or a private rerun, but it is not the frozen AVTime-Bench split:

python3 -m data.split_avtime_50k \
  --source data/AVTime-50K/all.jsonl \
  --output-root outputs/avtime-50k-derived-split \
  --derive-test-ids \
  --test-size 300 \
  --seed AVTime-50K-public-split-v1

Derived selection uses a 5 x 5 duration/event-count stratification and seeded SHA-256 ranking. Changing the seed changes the selected IDs while preserving the deterministic procedure.

Use --expected-rows 0 only when deliberately processing a different corpus version. Disabling the size check makes accidental corpus drift harder to detect.

2. AVCut: clipping complete videos

AVCut does not cut on scene changes alone. It fuses three signals:

  • ASR sentence ends and pauses;
  • visual scene transitions;
  • audio event onsets/offsets for music and sound effects.

Candidate boundaries are clustered, shifted away from the middle of speech toward pause anchors, filtered by confidence and minimum-gap constraints, and supplemented with soft boundaries when a segment becomes too long. The final timeline is full-coverage and non-overlapping.

The published YAML defaults include:

Setting Default
ASR backend parakeet
scene threshold 24.0
minimum boundary gap 5.0 s
minimum segment duration 6.0 s
long-segment soft threshold 16.0 s
MP3 sidecar enabled

Review avcut_config.yaml before a large run. Although the MP3 field is configurable in the single-video renderer, the released batch validator and Qwen segmented-caption path both expect the audio sidecar. Keep render.write_mp3_sidecar: true for the standard batch pipeline.

Cut one video 

python3 data/caption_pipeline/src/pipeline/avcut.py \
  /absolute/path/to/input.mp4 \
  --config data/caption_pipeline/avcut_config.yaml \
  --output-root /absolute/path/to/outputs/caption_pipeline/videos

The resulting directory is keyed by the source video stem:

/absolute/path/to/outputs/caption_pipeline/videos/<video_stem>/
├── segments.json
└── clips/
    ├── clip_0001.mp4
    ├── clip_0001.mp3
    ├── clip_0002.mp4
    ├── clip_0002.mp3
    └── ...

segments.json records:

  • source video, measured duration, device, ASR backend/model/language, config, and output root;
  • detector candidate counts and timing statistics;
  • segment index, clip path, start/end/duration in seconds;
  • boundary provenance;
  • audio, visual, and joint event labels.

avcut.py --dry-run still runs detection and writes segmentation metadata but does not render normal captionable media clips. Use it to inspect boundaries, not as input to live captioning.

Cut a flat directory 

python3 data/caption_pipeline/src/pipeline/avcut_batch.py \
  /absolute/path/to/flat_source_videos \
  --output-root /absolute/path/to/outputs/caption_pipeline/videos \
  --config data/caption_pipeline/avcut_config.yaml \
  --max-jobs 4

Batch discovery is intentionally non-recursive: only files directly inside the input directory are scanned. Supported extensions are .mp4, .m4v, .mov, .mkv, .avi, and .webm. Two files with the same stem but different extensions are rejected because they would map to the same output directory.

Batch mode provides:

  • a SQLite job table;
  • source signatures based on file size and nanosecond modification time;
  • per-video staging directories;
  • output validation before an atomic staging-to-final rename;
  • stable BLAKE2 sharding;
  • bounded GPU/device worker pools;
  • resumable completed and failed states.

The batch dry run creates a dummy output layout to exercise orchestration. It does not validate real detector or renderer quality.

Single-node wrapper

The wrapper derives parallelism from visible GPUs. Its default is four jobs per GPU:

bash data/caption_pipeline/src/pipeline/cut.sh \
  --input-root /absolute/path/to/flat_source_videos \
  --output-root /absolute/path/to/outputs/caption_pipeline/videos \
  --config data/caption_pipeline/avcut_config.yaml \
  --jobs-per-gpu 4

Always pass explicit input and output roots. This avoids ambiguity from machine-specific working directories and makes the run record portable.

Multi-node wrapper

The wrapper detects common scheduler rank variables. When a scheduler does not export them, set the AVTime-specific values:

AVCUT_NODE_RANK=0 AVCUT_NUM_NODES=2 \
bash data/caption_pipeline/src/pipeline/cut_multinode.sh \
  --input-root /absolute/path/to/flat_source_videos \
  --output-root /shared/path/to/outputs/caption_pipeline/videos \
  --config data/caption_pipeline/avcut_config.yaml

Run the command on every node with a different AVCUT_NODE_RANK. By default, only local rank zero on each node performs work. Each rank receives an independent state database and staging root.

The multi-node wrapper enables terminal-state skipping by default. A job marked completed or failed for the same source signature may be skipped without revalidating the final media. Inspect per-rank databases/logs before treating a distributed run as complete.

Detector degradation

AVCut warns and disables a detector channel when optional NeMo/WhisperX, PySceneDetect, or PANNs dependencies are unavailable. That behavior permits partial diagnostics, but it can degrade the cut plan to a small number of boundaries or even one full-video segment.

For production data construction, treat warnings about missing detectors or a missing PANNs checkpoint as failed environment validation, even when the process itself continues.

3. Segmented captioning

The caption stage consumes a root directory whose direct child directories each contain segments.json and rendered clips:

videos_root/
├── video_a/segments.json
├── video_a/clips/...
├── video_b/segments.json
└── video_b/clips/...

The Qwen path sends the clip video and its MP3 sidecar as separate local media items. It sets use_audio_in_video=false so that the audio sidecar, rather than audio embedded in the MP4, is the explicit audio source.

The caption prompt asks for one to three visual-first events per clip. Audio provides supporting evidence. For each successful source video, the script:

  1. requires a non-empty event array with numeric timestamps and non-empty captions;
  2. clamps every timestamp into the clip duration;
  3. folds a non-positive interval into the previous caption when possible (or expands the first such item to the clip), and pins only the first start and last end to the clip boundaries;
  4. shifts clip-local seconds by the clip start time;
  5. atomically writes one complete-video captions_merged.json.

The normalizer does not generally sort events or close every internal overlap/gap, so malformed chronology remains a generation-quality issue that should be audited.

The merged schema is:

[
  {
    "start_sec": 0.0,
    "end_sec": 5.84,
    "caption": "A person opens a door while footsteps are audible."
  }
]

Published Qwen caption-client defaults:

Setting Default
request model name endpoint fallback: Qwen/Qwen3-Omni-30B-A3B-Thinking; managed mode: served name or checkpoint basename
temperature / top-p / top-k 0.6 / 0.95 / 20
repetition penalty 1.0
maximum response tokens 32,768
video sampling 4 fps
concurrent video jobs 8
total in-flight clips 64
retries / request timeout 5 / 300 seconds

Option A: use an existing OpenAI-compatible endpoint

The endpoint must be able to read the exact same filesystem paths because the client sends file:// URIs:

python3 data/caption_pipeline/src/pipeline/qwen3_caption.py \
  /absolute/path/to/outputs/caption_pipeline/videos \
  --base-url http://127.0.0.1:8000/v1 \
  --model Qwen/Qwen3-Omni-30B-A3B-Thinking \
  --max-jobs 8 \
  --max-inflight-clips 64

Use repeated --source-video /absolute/path/to/original.mp4 arguments to select jobs by the exact segments.json input_video value. Use --video-fps, --video-min-frames, and --video-max-frames to control multimodal preprocessing.

Option B: let the client launch local vLLM 

python3 data/caption_pipeline/src/pipeline/qwen3_caption.py \
  /absolute/path/to/outputs/caption_pipeline/videos \
  --model-path /absolute/path/to/Qwen3-Omni-30B-A3B-Thinking \
  --tensor-parallel-size 8 \
  --max-jobs 8 \
  --max-inflight-clips 64

The managed-server mode starts vLLM, waits for readiness, runs caption jobs, and records server/raw-response logs. HTTP 400 and invalid-media responses are treated as deterministic errors and are not retried. Transient errors use the configured retry budget.

Qwen resume state is video-directory-level: captions_merged.json is written only after every clip in a video succeeds. The state signature hashes segments.json, not the clip bytes. If clip media changes without changing segments.json, delete or invalidate the corresponding state/output manually.

Multi-node segmented captioning 

AVCUT_NODE_RANK=0 AVCUT_NUM_NODES=2 \
bash data/caption_pipeline/src/pipeline/qwen3_caption_multinode.sh \
  /shared/path/to/outputs/caption_pipeline/videos \
  --model-path /shared/path/to/Qwen3-Omni-30B-A3B-Thinking \
  --tensor-parallel-size 8

Run once per node. Stable sharding ensures that the same video maps to the same rank for a fixed world size. The wrapper defaults to --keep-going, so a run can exit successfully even when individual videos failed. Completion must be checked from the state databases or the progress command:

Unlike the single-node client defaults, the multi-node wrapper supplies --max-jobs 128 --max-inflight-clips 128 unless explicitly overridden.

python3 data/caption_pipeline/src/pipeline/qwen3_caption_progress.py \
  /shared/path/to/outputs/caption_pipeline/videos

The monitor opens SQLite databases read-only and reports total, completed, running, failed, pending, and untracked jobs.

Gemini fallback

Gemini is a per-video-directory fallback. It uploads MP4 clips one at a time, writes recoverable per-clip JSON, and then merges the video:

export GEMINI_API_KEY=...
python3 data/caption_pipeline/src/pipeline/gemini_caption.py \
  /absolute/path/to/outputs/caption_pipeline/videos/video_a/segments.json \
  --model gemini-3.1-pro-preview

GOOGLE_API_KEY is also accepted. Keys are never read from repository files. Gemini writes:

video_a/
├── captions/
│   ├── clip_0001.json
│   └── ...
└── captions_merged.json

Its resume granularity is one clip, which is finer than the Qwen path. The Gemini normalizer is also intentionally looser; for comparable dataset construction, keep the backend and normalization path fixed across the run.

Part II: model training

4. Supervised fine-tuning (SFT)

Build the one-round dataset

The SFT builder accepts:

  • an optional LongVALE training JSONL;
  • AVTime training JSONL;
  • a deterministic AVTime sample count.

To build the intended mixed recipe from the currently supplied LongVALE file and a deterministic 1,500-row AVTime sample:

python3 -m training.sft.build_sft_dataset \
  --longvale-jsonl data/LongVALE/train_agent.jsonl \
  --avtime-jsonl outputs/avtime-50k-split/train.jsonl \
  --avtime-sample-count 1500 \
  --seed 20260702 \
  --output-jsonl outputs/training/sft_train.jsonl \
  --report-json outputs/training/sft_train.report.json

The builder consumes every valid row in the supplied LongVALE file; it does not assert a hard-coded LongVALE row count. Record the input hash and report instead of assuming a count from an earlier private snapshot.

AVTime-only SFT:

python3 -m training.sft.build_sft_dataset \
  --avtime-jsonl outputs/avtime-50k-split/train.jsonl \
  --avtime-sample-count 1500

Use all valid AVTime rows:

python3 -m training.sft.build_sft_dataset \
  --avtime-jsonl outputs/avtime-50k-split/train.jsonl \
  --avtime-sample-count 0

The builder converts percentage- or millisecond-based source events into one strict answer:

From 00 to 12, a person enters the room
From 13 to 27, the person sits at a table

Every output row has one system message, one user message, and one assistant message. Events are validated, de-duplicated, chronologically sorted, and written with one event per line. Rows without usable media or events are skipped; missing IDs or duplicate IDs are treated as errors. The JSON report records counts, source statistics, prompt text, and sample rows.

Launch SFT 

MODEL=/absolute/path/to/Qwen3-Omni-base \
DATASET=outputs/training/sft_train.jsonl \
OUTPUT_DIR=outputs/training/sft-checkpoints \
NNODES=4 \
NPROC_PER_NODE=8 \
NODE_RANK=0 \
MASTER_ADDR=trainer-0 \
bash training/sft/train_sft.sh

Run the same command on all four nodes, changing NODE_RANK to 0, 1, 2, or 3. The public defaults are:

Category Default
topology 4 nodes x 8 processes
TP / PP / CP / EP / ETP 2 / 1 / 2 / 8 / 1
global / micro batch 64 / 1
epochs 1
max sequence length 65,536
LoRA rank / alpha / dropout 128 / 256 / 0.05
trainable modules LLM LoRA; vision and aligner frozen
learning rate / minimum 5e-5 / 5e-6
held-out split ratio 0.002
save / evaluation interval 20 steps

After building the dataset, print the fully expanded command without launching training (the launcher validates that DATASET exists before its dry-run branch):

DRY_RUN=1 \
MODEL=/absolute/path/to/Qwen3-Omni-base \
DATASET=outputs/training/sft_train.jsonl \
bash training/sft/train_sft.sh

If Megatron-SWIFT is installed somewhere nonstandard, set MEGATRON_SFT_ENTRYPOINT. Use PYTHON_BIN to select the environment that contains both Swift and PyTorch.

5. Build the pure BiTSC GRPO dataset

Convert the one-round SFT rows:

python3 -m training.bitsc.build_grpo_dataset \
  --input-jsonl outputs/training/sft_train.jsonl \
  --output-jsonl outputs/training/bitsc_grpo.jsonl \
  --report-json outputs/training/bitsc_grpo.report.json \
  --seed 20260729

Add --check-media --media-root /absolute/media/root when media references should be verified during conversion.

The converter is deliberately strict:

  • exactly one assistant message;
  • exactly one video reference;
  • assistant output made only of valid From XX to YY, caption lines;
  • chronological events;
  • schema avtime_btsc_grpo_v1;
  • task exactly bitsc.

The solution field is a compact JSON string containing the complete reference event list. The default builder shuffles deterministically. Use --no-shuffle only for a controlled diagnostic.

Neither the dataset builder nor the reward plugin generates SC (segment-captioning) or TVG (temporal video grounding) tasks.

6. Run pure BiTSC GRPO

What “pure BiTSC” means here

The only registered reward name is avtime_btsc. The ORM rejects another schema or task with reward -1. It contains no reward branch for:

  • SC;
  • TVG;
  • data source identity;
  • caption style;
  • preferred event count;
  • task-specific formatting beyond the shared timestamp-caption contract.

The two BiTSC directions are:

  1. Time-to-event (T2E). At deterministic sampled times, find all active reference and predicted events. The caption-only judge scores semantic pairs from 0 to 4. Code normalizes these scores to [0, 1], performs maximum-weight bipartite matching, and computes 2 * matched_weight / (reference_count + prediction_count). A probe with only one active side scores zero; an all-empty probe is skipped.
  2. Event-to-time (E2T). Deterministically sample reference-event anchors. The judge selects all predicted captions describing the same event. Code takes the union of their time intervals and computes temporal union-IoU against the reference interval.

The final reward is the normalized weighted combination:

reward = w_t2e * mean(T2E probes) + w_e2t * mean(E2T anchors)

The defaults are w_t2e = 0.5 and w_e2t = 0.5.

Only caption text is sent to the semantic reward model. Timestamps remain in Python, so the language model cannot invent the temporal score. Probe and anchor sampling are deterministic for a fixed solution. Completions sharing the same reference solution are grouped into one judge request.

If RLAIF is disabled or a judge call fails, the ORM uses a deterministic token-F1 semantic fallback. This keeps training alive but is not equivalent to a successful model-based semantic judgment; monitor the BTSC-RLAIF logs.

Start the rollout service

On the rollout host:

ACTOR_MODEL=/absolute/path/to/sft-actor \
PORT=8100 \
TP=8 \
bash training/bitsc/serve_rollout.sh

Script defaults when not overridden:

  • port 8100;
  • TP 1;
  • GPU memory utilization 0.88;
  • maximum model length 34,816;
  • maximum concurrent sequences 64;
  • prefix caching enabled.

Start the caption-semantic reward service

On a separate reward host:

REWARD_MODEL=/absolute/path/to/reward-model \
SERVED_MODEL_NAME=reward-model \
PORT=8000 \
TP=8 \
bash training/bitsc/serve_reward_model.sh

Defaults are host 0.0.0.0, port 8000, TP 1, GPU memory utilization 0.90, maximum model length 32,768, maximum 32 sequences, and 32,768 batched tokens.

Use network controls appropriate for the cluster. The vLLM service should not be exposed to an untrusted network.

Launch GRPO 

ACTOR_MODEL=/absolute/path/to/sft-actor \
DATASET=outputs/training/bitsc_grpo.jsonl \
OUTPUT_DIR=outputs/training/bitsc-grpo-checkpoints \
ROLLOUT_HOST=rollout-host \
ROLLOUT_PORT=8100 \
REWARD_HOST=reward-host \
REWARD_PORT=8000 \
NNODES=2 \
NPROC_PER_NODE=8 \
NODE_RANK=0 \
MASTER_ADDR=trainer-0 \
bash training/bitsc/train_grpo.sh

Run on both training nodes with NODE_RANK=0 and NODE_RANK=1. The launcher checks that:

  • world size is divisible by TP * PP * CP;
  • EP divides the resulting data-parallel size;
  • global batch size is divisible by the number of generations.

Public GRPO defaults:

Category Default
topology 2 nodes x 8 processes
TP / PP / CP / EP / ETP 2 / 2 / 2 / 2 / 1
global / micro batch 128 / 1
generations per prompt 8
train iterations 100
max input / completion length 32,768 / 2,048
temperature / top-p / top-k 0.9 / 0.95 / 50
LoRA rank / alpha / dropout 64 / 128 / 0.05
learning rate / minimum 2e-6 / 2e-7
reward function avtime_btsc only

Useful ORM environment variables:

Variable Default Meaning
BTSC_PROBE_COUNT 8 deterministic T2E probe count
BTSC_ANCHOR_COUNT 4 reference anchors for E2T
BTSC_TIME_TO_EVENT_WEIGHT 0.5 T2E weight
BTSC_EVENT_TO_TIME_WEIGHT 0.5 E2T weight
BTSC_RLAIF_ENABLED 1 enable semantic model
BTSC_RLAIF_FRACTION 1.0 deterministic fraction sent to judge
BTSC_RLAIF_MAX_CONCURRENCY 8 concurrent judge groups
BTSC_RLAIF_TIMEOUT_SEC 120 request timeout
BTSC_RLAIF_MAX_TOKENS 4096 judge response ceiling
BTSC_RLAIF_JSON_OBJECT 1 request strict JSON schema
BTSC_REWARD_BASE_URL http://127.0.0.1:8000/v1 OpenAI-compatible endpoint
BTSC_REWARD_MODEL reward-model served model name

After building the GRPO dataset, inspect the expanded command (the launcher validates that DATASET exists before its dry-run branch):

DRY_RUN=1 \
ACTOR_MODEL=/absolute/path/to/sft-actor \
DATASET=outputs/training/bitsc_grpo.jsonl \
bash training/bitsc/train_grpo.sh

Part III: inference and evaluation

7. Run LongVALE-compatible inference

This repository includes the LongVALE-compatible dense video captioning inference protocol and the pinned official adjacent-caption merge port. It does not redistribute the full third-party LongVALE evaluator.

“LongVALE-compatible” in this release means:

  • the frozen training-style prompt;
  • one output line per event: From XX to YY, caption;
  • percentage timestamps in [0, 99];
  • official Qwen multimodal preprocessing when explicitly selected;
  • the pinned adjacent-caption merge used by the canonical AVTime path.

Prepare reference-free inference JSONL

Each row must contain one video and a stable video ID. The AVTime target-input builder produces this contract automatically. For another LongVALE-compatible set, create rows equivalent to:

{
  "id": "record_0001",
  "tools": [],
  "videos": ["/absolute/path/to/video_0001.mp4"],
  "messages": [
    {"role": "system", "content": "placeholder"},
    {"role": "user", "content": "<video>"}
  ],
  "metadata": {
    "video_id": "video_0001",
    "duration_ms": 120000
  }
}

With --prompt-style train, the frozen training prompt replaces the row text; the direct inference runner does not consume the sample messages in that mode. They are shown to mirror the stricter AVTime target-input structure. With --prompt-style jsonl, the direct runner requires at least one user message whose content can be extracted; a system message is optional. In all cases, keep references and assistant messages out of inference inputs.

Run one LongVALE-style inference job 

python3 -m benchmark.infer_qwen3_omni \
  --jsonl /absolute/path/to/longvale_target_inputs.jsonl \
  --model-path /absolute/path/to/target-model \
  --model-name avtime-model \
  --output-dir outputs/longvale/inference \
  --predictions-jsonl outputs/longvale/raw_predictions.jsonl \
  --prompt-style train \
  --qwen-omni-media-mode official \
  --use-audio-in-video \
  --temperature 0 \
  --top-p 1 \
  --top-k 20 \
  --repetition-penalty 1 \
  --max-completion-tokens 8192 \
  --batch-size 8 \
  --vllm-tensor-parallel-size 4 \
  --vllm-max-model-len 65536

The direct inference CLI defaults to resumable and continue-on-error=true. That option continues with later rows after a row-level failure; it does not turn an incomplete run into success. The process still exits with status 2 when any row failed. For immediate fail-closed behavior, add:

--no-resume --no-continue-on-error

Apply the pinned official adjacent-caption merge

Normalization requires target identities, raw answers, a model identity, and an explicit decoding record:

python3 -m benchmark.normalize_predictions \
  --target-inputs /absolute/path/to/longvale_target_inputs.jsonl \
  --raw outputs/longvale/raw_predictions.jsonl \
  --output outputs/longvale/predictions.jsonl \
  --failures outputs/longvale/prediction_failures.jsonl \
  --report outputs/longvale/prediction_report.json \
  --model-id avtime-model \
  --model-path /absolute/path/to/target-model \
  --protocol longvale_train_dvc_v1 \
  --decoding-json '{"temperature":0,"top_p":1,"top_k":20,"max_tokens":8192}'

The parser ignores non-protocol prose with recorded warnings, rejects a result with no valid event lines, sorts valid events by time, and merges adjacent captions only when:

SequenceMatcher(caption_a, caption_b).ratio() > 0.98
and next.start_pct - current.end_pct is in [-1, 1]

The report records the pinned upstream LongVALE commit, upstream file hash, merge thresholds, and frozen validation-vector hash. Use the official LongVALE project separately when full LongVALE benchmark metrics are required.

8. Run AVTime-Bench

Canonical HQ300 protocol

benchmark/run_avtime_bench.sh is the canonical entry point. It executes four stages:

  1. build reference-free target inputs if absent;
  2. run two target-model shards and enforce exact 300-video coverage;
  3. normalize with the LongVALE line protocol and official adjacent-caption merge;
  4. prepare caption-only requests, run Qwen3.6-27B, and deterministically score six AVTime metrics.

Expected private files under data/AVTime-Bench/:

File Purpose
test.jsonl source test rows
inventory.jsonl audited IDs, durations, hashes, and absolute media paths
benchmark_reference_manifest.jsonl private scoring references
target_inputs.jsonl generated reference-free model inputs

If target_inputs.jsonl is absent, the wrapper writes it under BENCH_ROOT. Therefore that directory must be writable, or TARGET_INPUTS must point to a writable alternative.

Build inputs explicitly:

python3 -m benchmark.build_target_inputs \
  --test-jsonl data/AVTime-Bench/test.jsonl \
  --inventory data/AVTime-Bench/inventory.jsonl \
  --output data/AVTime-Bench/target_inputs.jsonl \
  --report data/AVTime-Bench/target_inputs_report.json

The builder requires:

  • unique inventory stable_id values;
  • unique benchmark video_id values;
  • an existing absolute resolved_media_path for every row;
  • exactly one system and one user message;
  • no extra fields inside those message objects;
  • full test/inventory coverage.

Assistant/reference content is removed. Output metadata retains audited duration and source hashes.

Run the complete benchmark

The default topology expects eight GPUs. Target inference runs as two parallel TP=4 shards, then the judge reuses eight GPUs with TP=8:

TARGET_MODEL=/absolute/path/to/target-model \
TARGET_MODEL_ID=avtime-model \
JUDGE_MODEL=Qwen/Qwen3.6-27B \
JUDGE_MODEL_ID=Qwen3.6-27B-avtime-judge-v1 \
RUN_ID=avtime-model-hq300-qwen36-27b \
bash benchmark/run_avtime_bench.sh

Important canonical defaults:

Target setting Default
expected videos 300
GPU groups 0,1,2,3 and 4,5,6,7
tensor parallelism per shard 4
rows per shard 150 and 150
batch size per replica 512
max model length 65,536
max completion tokens 8,192
temperature / top-p / top-k 0 / 1 / 20
repetition penalty 1
seed 1234
prompt frozen LongVALE training prompt
media preprocessing official Qwen path
audio in video enabled
target resume disabled
target continue-on-error disabled

Useful hardware overrides:

TARGET_MODEL=/absolute/path/to/target-model \
TARGET_SHARD0_GPUS=0,1 \
TARGET_SHARD1_GPUS=2,3 \
TARGET_TP=2 \
TARGET_BATCH_SIZE=32 \
TARGET_MAX_NUM_SEQS=8 \
TARGET_GPU_MEMORY_UTILIZATION=0.80 \
JUDGE_MODEL=/absolute/path/to/Qwen3.6-27B \
JUDGE_TP=4 \
JUDGE_MAX_MODEL_LEN=32768 \
JUDGE_BATCH_SIZE=16 \
RUN_ID=hardware-adjusted-run \
bash benchmark/run_avtime_bench.sh

Changing topology or batch size does not change the intended semantic protocol, but every override must be recorded with the resulting run artifacts.

The wrapper refuses to reuse an existing OUTPUT_ROOT. Choose a new RUN_ID or an empty OUTPUT_ROOT for every run. It also fails when either target shard returns a nonzero status or produces anything other than exact expected line coverage.

Qwen3.6-27B judge behavior

The judge receives caption text plus a modality rubric; it does not receive timestamps. It serves three request kinds:

  • t2e_scalar: semantic score for captions active at a fixed point;
  • semantic_selector: prediction indices matching one reference event;
  • semantic_pair_selector: two selector sets for temporal relation scoring.

The first canonical judge pass uses 512 output tokens and batch size 32. If that process exits nonzero, the wrapper retries with 4,096 tokens and batch size 16. Resume reuse is allowed only when the query ID, axis, branch, video ID, request type, model ID, prompt version and hash, schema version and hash, and request hash all match.

Six AVTime scores

The summary contains exactly:

Direction Audio Visual
Time-to-event T2E-A T2E-V
Event-to-time E2T-A E2T-V
Event-to-event E2E-A E2E-V

Ownership is intentionally separated:

  • Qwen3.6-27B judges semantic equivalence or selects caption indices.
  • Deterministic Python converts percentages to intervals, finds active events, unions intervals, computes tIoU, and evaluates before/after/overlap relations.

For T2E, a query with no active prediction deterministically scores zero and does not call the judge. E2T uses union-tIoU over all semantically selected prediction intervals. E2E derives the relation between two selected interval unions with the benchmark tolerance. Missing or invalid semantic responses are fatal; there is no lexical fallback in benchmark scoring.

Canonical output tree 

outputs/avtime-bench/<RUN_ID>/
├── target_inputs_report.json            # when inputs were built
├── inference/
│   ├── shard0/
│   │   ├── raw_predictions.jsonl
│   │   └── summary.json
│   └── shard1/
│       ├── raw_predictions.jsonl
│       └── summary.json
├── predictions.jsonl
├── prediction_failures.jsonl
├── prediction_report.json
├── semantic_requests.jsonl
├── semantic_responses.jsonl
├── semantic_failures.jsonl
├── semantic_report.json
└── eval/<RUN_ID>/
    ├── run_config.json
    ├── t2e_trace.jsonl
    ├── e2t_trace.jsonl
    ├── e2e_trace.jsonl
    └── summary.json

Structured-JSON experimental path

The earlier structured-output experiment remains available:

TARGET_MODEL=/absolute/path/to/target-model \
TARGET_MODEL_ID=avtime-model-structured \
JUDGE_MODEL=Qwen/Qwen3.6-27B \
RUN_ID=structured-experiment \
bash benchmark/run_avtime_bench_structured.sh

It is not the canonical main-table protocol.

Property Canonical wrapper Structured wrapper
target prompt frozen train prompt input JSONL messages by default
target output LongVALE lines strict JSON schema
media mode official Swift-compatible by default
execution two fail-closed shards one resumable pass
coverage gate exact 300 by default no canonical 300-row gate
target row errors abort immediately process later rows, then exit nonzero if any failed
judge retry pass included one pass
output-root protection refuses reuse scorer directory remains immutable

The structured target subprocess defaults to TP=1, batch size 8, 4,096 completion tokens, top-k -1, resumable inference, and row-level continue-on-error. Because the wrapper uses set -e, any final nonzero target status still stops the wrapper before normalize/judge/score. Its behavior and artifacts must not be described as a canonical HQ300 run.

Outputs and reproducibility

Time representation

Model-facing answers use inclusive integer percentage buckets from 00 to 99. When converted to milliseconds, bucket p covers its percentage cell:

start_ms = floor((start_pct / 100) * duration_ms)
end_ms   = ceil(((end_pct + 1) / 100) * duration_ms)

The resulting internal interval is half-open and clamped to the measured video duration. This makes From 00 to 00 a non-empty first bucket and From 99 to 99 reach the video end.

Caption-pipeline outputs use seconds directly. Training and benchmark normalizers perform explicit conversions; do not mix the two representations without the corresponding duration.

Run identity

For every publishable run, preserve:

  • repository commit;
  • input and output hashes;
  • model checkpoint identity and revision;
  • exact Python/CUDA/PyTorch/vLLM/Swift versions;
  • GPU model and count;
  • all non-default environment variables;
  • split manifest;
  • SFT and GRPO builder reports;
  • prediction and judge reports;
  • benchmark run_config.json and trace files.

Do not overwrite an old result with a new checkpoint or judge response.

Fail-closed versus keep-going stages

The repository uses different failure policies for different operational needs:

Stage Default behavior
AVTime split fail on malformed identity/coverage
AVCut batch record per-video state; resumable
Qwen caption client resumable per video
Qwen multi-node caption wrapper keep going; inspect DB
direct benchmark inference CLI resume + continue on error
canonical AVTime-Bench target inference no resume, fail closed
canonical benchmark scoring fail on missing judge response

Read process exit status together with reports and state databases. A zero exit from a keep-going wrapper does not prove complete coverage.

Validation

Install the development requirements and run:

python3 -m pytest -q
python3 scripts/check_release.py

The tests cover:

  • split and training data builders;
  • strict pure-BiTSC input validation;
  • ORM scoring, semantic fallback, and task isolation;
  • SFT/GRPO launcher expansion and topology checks;
  • target-input reference removal;
  • LongVALE parsing and pinned official merge behavior;
  • canonical wrapper flags and fail-closed coverage;
  • judge request contracts;
  • all six AVTime scoring branches.

Shell syntax can be checked without starting GPU work:

find data training benchmark -type f -name '*.sh' -print0 \
  | xargs -0 -n1 bash -n

Use launcher DRY_RUN=1 modes where provided to inspect resolved arguments. Unit tests and dry runs do not replace a small end-to-end media smoke test on the actual GPU environment.

Troubleshooting

AVCut produces one very long clip

Inspect warnings for unavailable ASR, scene, or PANNs detectors. Confirm the PANNs checkpoint path, NeMo/WhisperX installation, PySceneDetect, FFmpeg, and the selected CUDA device. A process that degraded gracefully is not necessarily a valid production cut.

Qwen captioning cannot open file:// media

The endpoint and client must see the same absolute paths. A containerized or remote endpoint needs the media directory mounted at the identical path. The released segmented captioner also expects MP3 sidecars.

A distributed caption job exits zero with failed videos

The multi-node wrapper enables keep-going. Run qwen3_caption_progress.py and inspect the per-rank SQLite databases and raw response logs.

SFT or GRPO entrypoint is not found

Activate the environment containing Megatron-SWIFT or set PYTHON_BIN and, when necessary, MEGATRON_SFT_ENTRYPOINT, MEGATRON_RLHF_ENTRYPOINT, or SWIFT_ROLLOUT_ENTRYPOINT explicitly.

GRPO topology is rejected

Check NNODES * NPROC_PER_NODE against TP * PP * CP, then check that EP divides the derived data-parallel size. Also make GLOBAL_BATCH_SIZE divisible by NUM_GENERATIONS.

BiTSC reward suddenly uses lexical fallback

Look for BTSC-RLAIF warnings. Verify the reward endpoint, served model name, timeout, response schema support, and network reachability. The token-F1 path keeps execution moving but should be reported as a degraded reward run.

AVTime-Bench refuses an output directory

Canonical run artifacts are immutable. Choose a new RUN_ID or OUTPUT_ROOT. Do not delete an old run merely to reuse its name.

The benchmark target stage fails coverage

Confirm that target_inputs.jsonl has exactly EXPECTED_VIDEOS lines and that each shard produced its exact assigned count. A raw error row still indicates failed inference and the canonical wrapper aborts on the originating process.

The judge retry still fails

Inspect semantic_failures.jsonl and semantic_report.json. The second pass uses a larger token ceiling and smaller batch, but it still requires every response to pass identity and schema validation. Benchmark scoring intentionally has no silent fallback.

Release and redistribution boundary

  • The frozen split ID list is included; the full dataset is not.
  • The code never requires credentials to be stored in tracked files.
  • This copy contains no absolute internal cluster paths or bundled secrets.
  • The official adjacent-caption merge is a small pinned compatibility port; the complete third-party LongVALE evaluator is not redistributed.
  • Dataset, checkpoint, and third-party software licenses must be reviewed independently before redistribution.
  • Complete RELEASE_CHECKLIST.md before changing repository visibility or publishing scores.
  • Repository-level extraction and provenance notes are recorded in PROVENANCE.md.

The public benchmark default is the requested Qwen3.6-27B judge. An older internal launcher used Qwen3.6-35B-A3B. Historical 35B-A3B results must not be labeled as 27B results; any released 27B number should come from a complete 27B judge rerun with its own immutable artifacts.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support