|
Download README_benchmark.md from ChatterjeeLab/SF-Cluster: direct link, hf CLI and curl.
- Browser
- Download file 4.07 kB
-
https://huggingface.co/ChatterjeeLab/SF-Cluster/resolve/main/README_benchmark.md
- Command line
-
hf download hf://ChatterjeeLab/SF-Cluster/README_benchmark.md
-
curl -L -o README_benchmark.md https://huggingface.co/ChatterjeeLab/SF-Cluster/resolve/main/README_benchmark.md
4.07 kB
| # SF-Cluster benchmark reproduction (CPU-only) | |
| Self-contained bundle that reproduces the headline **`minority_hit_rate`** table | |
| of SF-Cluster: how often each MSA-subsetting method recovers the *minority / | |
| rare* conformational state of a fold-switching protein, scored against locked | |
| reference structures. No GPU required — all AlphaFold2 predictions are shipped. | |
| ## What it reproduces | |
| Per (method arm x case) fraction of predictions that pass the §9.1 hit criterion | |
| for the minority state. Cases: **KaiB**, **GA98**, **GB98**. Arms: four SF | |
| methods (`mosaic_raw`, `gradient_raw`, `contrast_raw`, `region_cluster_raw`) plus | |
| the `afcluster` (AF-Cluster / DBSCAN) baseline. | |
| ### Expected table (published targets, tolerance +/-0.02) | |
| | arm | KaiB | GA98 | GB98 | | |
| |--------------------|--------|--------|--------| | |
| | mosaic_raw | 0.9500 | 0.9250 | 0.1875 | | |
| | gradient_raw | 0.6500 | 0.5000 | 0.5000 | | |
| | contrast_raw | 0.6750 | 0.4500 | 0.0750 | | |
| | region_cluster_raw | 0.5375 | 0.5000 | 0.4500 | | |
| | afcluster | 0.3625 | 0.4625 | 0.4375 | | |
| KaiB/GA98/GB98 SF arms are n=80 predictions each; afcluster KaiB is n=320, | |
| GA98/GB98 n=80. | |
| ## Hit criterion (protocol §9.1, all three must hold) | |
| For the minority reference state: | |
| 1. C-alpha RMSD <= 3.0 A on `common_core` residues (Biopython `Superimposer`). | |
| 2. Mean pLDDT >= 70 overall. | |
| 3. Mean pLDDT >= 70 in `switch_region` (3 A displacement region). | |
| The authoritative flag is the `<state>__hit_primary` column in `evals.tsv`. | |
| `minority_hit_rate` = mean of that column. Minority state per case: | |
| | Case | Minority-state column | Note | | |
| |------|----------------------------|------| | |
| | KaiB | `state_A_2QKE__hit_primary` | fold-switched 2QKE chain B | | |
| | GA98 | `GA98_2LHC__hit_primary` | alternate GA-fold basin | | |
| | GB98 | `GA98_2LHC__hit_primary` | GA98 fold is the minority conformation for GB98 | | |
| ## GA_GB CLI caveat | |
| The scorer's `--case` accepts only `KaiB`, `GA_GB`, `Mpt53`. **Both GA98 and GB98 | |
| are scored as `--case GA_GB`**; they differ only in which reference RMSD column is | |
| read. Consequently GB98's minority state is the `GA98_2LHC` column (the alternate | |
| fold), not its own native `GB98_2LHD` column. | |
| ## Dependencies | |
| Pip-only: `numpy`, `biopython`, `pyyaml`. No conda, no GPU. | |
| ```bash | |
| pip install numpy biopython pyyaml | |
| ``` | |
| `TMalign` is **optional**: if present on `PATH` it fills the `tmalign_*` columns; | |
| if absent those columns are `NA`. It is **not** part of the hit criterion, so the | |
| reproduction is unaffected. | |
| ## Commands | |
| Run from anywhere (bundle root is resolved relative to `reproduce_benchmark.py`): | |
| ```bash | |
| # Fast path: aggregate the shipped per-prediction evals.tsv and assert targets. | |
| python reproduce_benchmark.py --mode precomputed | |
| # Full path: re-run the scorer over the shipped PDBs, regenerate evals.tsv, | |
| # then aggregate and assert. Slower (parses ~1440 PDBs) but fully from scratch. | |
| python reproduce_benchmark.py --mode score | |
| ``` | |
| Both print the case x arm table and exit non-zero if any value is outside | |
| +/-0.02 of the published target. | |
| Ad-hoc aggregation of an arbitrary bench root: | |
| ```bash | |
| python eval/aggregate_hits.py --bench-root ./bench | |
| ``` | |
| ## Layout | |
| ``` | |
| reproduce_benchmark.py top-level driver (precomputed | score) | |
| README_benchmark.md this file | |
| eval/ | |
| evaluate_prediction.py per-prediction scorer (§9.1 hit_primary) | |
| batch_eval.py batch driver -> evals.tsv | |
| aggregate_hits.py evals.tsv -> minority_hit_rate table | |
| bench/ | |
| configs/cases.yaml case metadata (construct_start offsets) | |
| data/manifests/eval_regions.tsv common_core / switch_region residue lists | |
| data/processed/structures/ 4 reference PDBs (KaiB state_A/B, GA98, GB98) | |
| results/baseline_p12/<arm>/<case>/refine_per_state/ 80 PDBs + evals.tsv | |
| results/baseline/afcluster/<case>/refine_per_state/ PDBs + evals.tsv | |
| ``` | |
| The scorer resolves its data/config root from `SF_BENCH_ROOT` (set automatically | |
| by `reproduce_benchmark.py --mode score`), falling back to the repo layout. | |