Efficientformer for Multitask
Overview
This is an experimental Efficientformer codebase for Multitask. It keeps the tiny setup intentionally manageable so architecture changes can be inspected before a full training run.
Architecture
- Architecture: Efficientformer
- Scale: tiny
- Attention: standard
- Fusion: cross attention
- Activation: relu
- Normalization: batchnorm
Training details
- Weights initialized with a orthogonal scheme.
- Runs on a single consumer GPU.
- Full smoke test takes minutes, not hours.
- No distributed setup required.
Limitations
The initialization checkpoint has not been trained or audited for robustness, fairness, or domain transfer. The implementation should be treated as an experimental starting point. Results from a future trained checkpoint must be documented separately from the defaults shipped here.
Default experiment recipe
The included configuration uses lion with a linear warmup schedule. These are starting values in the script, not evidence of a completed run. For a meaningful evaluation, train all baselines with the same data exposure, tuning budget, and random seeds.
Quick check
Run python model.py --help for the built-in smoke test. Because this is a custom implementation, generic automatic loading APIs require an explicit adapter before use.
Roadmap
Next steps, in no particular order: a cleaner entry point, input-shape docs, and a small evaluation script if time permits.
Repository status
One Python module contains the model and a runnable example.
Generated configuration lives in config.json; the recipe in training_args.json.
The checkpoint is an initialization artifact and should not be read as a trained model.
Scores are deliberately absent.
Files
model.pyβ primary artifactREADME.mdβ this documentationconfig.jsonβ architecture configurationtraining_args.jsonβ default experiment settingsmodel.safetensorsβ initialization checkpoint
License
Released under apache-2.0. Review the source-data terms separately when this repository is used with external datasets.
- Downloads last month
- 3