| --- |
| language: |
| - en |
| datasets: |
| - RosettaCommons/PISCES-CulledPDB |
| license: mit |
| library_name: pytorch |
| base_model: facebook/esm2_t6_8M_UR50D |
| tags: |
| - biology |
| - bioinformatics |
| - protein-secondary-structure |
| - esm2 |
| - pytorch |
| - bilstm |
| pipeline_tag: token-classification |
| model-index: |
| - name: SERAPH |
| results: |
| - task: |
| type: token-classification |
| name: Secondary Structure Prediction (Q3) |
| metrics: |
| - name: Q3 Test Accuracy |
| type: accuracy |
| value: 75.31 |
| --- |
| |
| # SERAPH (Secondary Structure Recognition & Prediction Hub) |
|
|
| **SERAPH** is a deep learning model designed for 3-state (Q3) protein secondary structure prediction. It processes raw single amino acid sequences and predicts residue-level secondary structure states: **Alpha Helix (`H`)**, **Beta Sheet (`E`)**, or **Coil/Loop (`C`)**. |
|
|
| The model leverages a fine-tuned `facebook/esm2_t6_8M_UR50D` backbone combined with a 1D Convolutional feature extractor and a 2-layer Bidirectional LSTM to capture local motifs and long-range sequence context simultaneously. |
|
|
| ## Model Details |
|
|
| ### Model Description |
|
|
| - **Developed by:** Rogue Builds |
| - **Model Type:** Protein Language Model + Conv1D + BiLSTM |
| - **Language(s):** Protein Sequences (Amino Acid single-letter codes) |
| - **License:** MIT |
| - **Finetuned from model:** `facebook/esm2_t6_8M_UR50D` |
|
|
| ### Model Sources |
|
|
| - **Repository:** `PypCoder/SERAPH` |
|
|
| --- |
|
|
| ## Intended Uses & Limitations |
|
|
| ### Direct Use |
| * Residue-level 3-state (Q3) protein secondary structure prediction. |
| * Single-sequence inference when Multiple Sequence Alignment (MSA) generation is computationally prohibitive or unavailable. |
| * Integration into downstream bioinformatics analysis pipelines and structural annotation tools. |
|
|
| ### Out-of-Scope & Misuse |
| * **3D Coordinate Generation**: SERAPH predicts 1D structural states (`H`, `E`, `C`), not 3D atomic coordinates. |
| * **Q8 DSSP Prediction**: The model is trained strictly for 3-state classification and does not differentiate between 8-state DSSP assignments (e.g., distinguishing $3_{10}$-helices from $\alpha$-helices). |
| |
| ### Known Limitations |
| * **Sequence Length Limit**: Input sequences are capped at **512 tokens** due to the positional encoding window of the underlying ESM-2 backbone. |
| * **Single-Sequence Bias**: Lacks explicit MSA input features; evolutionary context is derived solely from pre-trained ESM-2 representations. |
| |
| --- |
| |
| ## How to Get Started |
| |
| ### Prerequisites |
| |
| ```bash |
| pip install torch transformers huggingface_hub |
| ``` |
| |
| ### Python Inference Example |
| |
| ```python |
| import torch |
| import torch.nn as nn |
| from transformers import EsmModel, EsmTokenizer |
|
|
| # 1. Define SERAPH Architecture |
| class SERAPH(nn.Module): |
| def __init__(self, esm_model, conv_channels=256, kernel_size=7, lstm_hidden=256, num_classes=3, dropout=0.3): |
| super().__init__() |
| self.esm = esm_model |
| esm_embed_dim = self.esm.config.hidden_size |
| self.conv = nn.Conv1d(esm_embed_dim, conv_channels, kernel_size=kernel_size, padding=kernel_size // 2) |
| self.bn = nn.BatchNorm1d(conv_channels) |
| self.dropout = nn.Dropout(dropout) |
| self.bilstm = nn.LSTM(conv_channels, lstm_hidden, num_layers=2, batch_first=True, bidirectional=True) |
| self.fc = nn.Linear(lstm_hidden * 2, num_classes) |
| |
| def forward(self, input_ids, attention_mask=None): |
| x = self.esm(input_ids=input_ids, attention_mask=attention_mask).last_hidden_state |
| x = x.transpose(1, 2) |
| x = torch.relu(self.bn(self.conv(x))) |
| x = self.dropout(x) |
| x = x.transpose(1, 2) |
| x, _ = self.bilstm(x) |
| x = self.dropout(x) |
| return self.fc(x) |
| |
| # 2. Load Tokenizer & Base Backbone |
| ESM_MODEL_ID = "facebook/esm2_t6_8M_UR50D" |
| tokenizer = EsmTokenizer.from_pretrained(ESM_MODEL_ID) |
| esm_backbone = EsmModel.from_pretrained(ESM_MODEL_ID) |
|
|
| model = SERAPH(esm_model=esm_backbone) |
|
|
| # Load weight checkpoint |
| # checkpoint = torch.load("SERAPH.pth", map_location="cpu") |
| # model.load_state_dict(checkpoint["model_state_dict"]) |
| model.eval() |
| |
| # 3. Perform Prediction |
| IDX_TO_LABEL = {0: 'H', 1: 'E', 2: 'C'} |
| sequence = "MVLSPADKTNVKAAWGKVGAHAGEYGAEALERMFLSFPTTKTYFPHFDLSHGSAQVKGHGKKVADALTNAVAHVDDMPNALSALSDLHAHKLRVDPVNFKLLSHCLLVTLAAHLPAEFTPAVHASLDKFLASVSTVLTSKYR" |
| |
| tokens = tokenizer(sequence, return_tensors="pt", truncation=True, max_length=512) |
| |
| with torch.no_grad(): |
| output = model(input_ids=tokens["input_ids"], attention_mask=tokens["attention_mask"]) |
| preds = output.argmax(dim=-1)[0] |
| |
| # Omit special tokens [CLS] and [EOS] |
| prediction = "".join([IDX_TO_LABEL[p.item()] for p in preds[1:-1]]) |
| print(f"Sequence: {sequence}") |
| print(f"Prediction: {prediction}") |
| ``` |
| |
| --- |
| |
| ## Training Details |
| |
| ### Training Data |
| |
| * **Dataset**: CullPDB (~6,000 non-redundant protein chains). |
| |
| ### Training Procedure |
| |
| * **Optimizer**: Adam (`lr=5e-5`, `weight_decay=1e-4`) |
| * **Loss Function**: `CrossEntropyLoss` with class weight adjustments `[H: 1.3, E: 1.3, C: 1.0]` |
| * **Gradient Clipping**: `max_norm = 1.0` |
| * **Scheduler**: `ReduceLROnPlateau` (`patience=3`, `factor=0.5`) |
| * **Batch Size**: 32 (with dynamic sequence padding) |
| * **Epochs**: 15 |
| * **Backbone Unfreezing**: Top 2 transformer layers of `facebook/esm2_t6_8M_UR50D` unfrozen during training. |
| |
| ### Parameter Distribution |
| |
| | Layer Component | Trainable Parameters | |
| |---|---| |
| | ESM-2 Backbone (Unfrozen layers) | ~2,600,000 | |
| | Conv1D (`320 → 256`, `k=7`) | 573,440 | |
| | BatchNorm1d (`256`) | 512 | |
| | BiLSTM (2 Layers, hidden=256) | ~1,311,232 | |
| | Linear Head (`512 → 3`) | 1,539 | |
| | **Total Trainable Parameters** | **3,205,379** | |
| |
| --- |
| |
| ## Evaluation Results |
| |
| ### Evaluation Benchmark |
| |
| Evaluated on the standard **CB513** benchmark dataset. |
| |
| ### Metrics |
| |
| | Evaluation Metric | Score | |
| |---|---| |
| | **Q3 Test Accuracy (CB513)** | **75.31%** | |
| | **Q3 Training Accuracy** | **79.34%** | |
| |
| #### Class Breakdown |
| |
| | Structure Class | Precision | Recall | |
| |---|---|---| |
| | **Helix (`H`)** | 0.82 | 0.80 | |
| | **Sheet (`E`)** | 0.63 | 0.81 | |
| | **Coil (`C`)** | 0.79 | 0.68 | |
| |
| --- |
| |
| ## Citation & Contact |
| |
| If you use SERAPH in your work, please cite the underlying ESM-2 paper and reference this repository: |
| |
| ```bibtex |
| @software{seraph2026, |
| author = {Muhammad Asad Ullah}, |
| title = {SERAPH: Secondary Structure Recognition & Prediction Hub}, |
| year = {2026}, |
| url = {https://huggingface.co/PypCoder/SERAPH} |
| } |
| ``` |
| |
