Efficientformer for Multitask

Overview

This is an experimental Efficientformer codebase for Multitask. It keeps the tiny setup intentionally manageable so architecture changes can be inspected before a full training run.


Architecture

  • Architecture: Efficientformer
  • Scale: tiny
  • Attention: standard
  • Fusion: cross attention
  • Activation: relu
  • Normalization: batchnorm

Training details

  • Weights initialized with a orthogonal scheme.
  • Runs on a single consumer GPU.
  • Full smoke test takes minutes, not hours.
  • No distributed setup required.

Limitations

The initialization checkpoint has not been trained or audited for robustness, fairness, or domain transfer. The implementation should be treated as an experimental starting point. Results from a future trained checkpoint must be documented separately from the defaults shipped here.


Default experiment recipe

The included configuration uses lion with a linear warmup schedule. These are starting values in the script, not evidence of a completed run. For a meaningful evaluation, train all baselines with the same data exposure, tuning budget, and random seeds.


Quick check

Run python model.py --help for the built-in smoke test. Because this is a custom implementation, generic automatic loading APIs require an explicit adapter before use.


Roadmap

Next steps, in no particular order: a cleaner entry point, input-shape docs, and a small evaluation script if time permits.


Repository status

One Python module contains the model and a runnable example. Generated configuration lives in config.json; the recipe in training_args.json. The checkpoint is an initialization artifact and should not be read as a trained model. Scores are deliberately absent.


Files

  • model.py β€” primary artifact
  • README.md β€” this documentation
  • config.json β€” architecture configuration
  • training_args.json β€” default experiment settings
  • model.safetensors β€” initialization checkpoint

License

Released under apache-2.0. Review the source-data terms separately when this repository is used with external datasets.

Downloads last month
3
Safetensors
Model size
33.1k params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support