ESMFold2 has high pLDDT in disordered regions

[agitter]

ESMFold2 reports high pLDDT in disordered regions of proteins. I started to explore this with disordered proteins from DisProt and collected examples in https://github.com/agitter/esmfold2-disordered. Here is an illustration using O94687.

AlphaFold2:

AlphaFold3:

ESMFold2:

pLDDT comparison:

Another independent example from Ayushman Mallick.

[Ayushmania2002]

---

Title: ESMFold2 API pLDDT significantly higher than local HuggingFace weights for IDR-containing proteins

---

Environment

• Model: esmfold2-2026-05 (Full)
• API endpoint: forge.biohub.ai
• Local weights: biohub/ESMFold2 (HuggingFace)
• ESM package: git+https://github.com/Biohub/esm.git@c94ed8d
• Protein: OsIPI1 (LOC_Os01g24880.1), Oryza sativa RING-type E3 ubiquitin ligase, 455 aa, ~88% intrinsically disordered

---

Summary

We observed a large and reproducible discrepancy in pLDDT scores between the Biohub API (forge.biohub.ai) and local inference using the HuggingFace weights (biohub/ESMFold2) for the same protein, same model, same parameters, and same MSA. The API consistently returns pLDDT ~0.18 higher than local inference across all conditions tested.

This was flagged on Twitter by @GregPreibisch, who independently ran the same sequence locally using the HuggingFace weights and delivered results to us, enabling the direct comparison.

---

Protein sequence

>OsIPI1_LOC_Os01g24880.1 (455 aa)
MGAEEEEEPASAVGREGGGGGGGARAAGAGAGGDTADDDDSGESAAAVVPCSICLDAVVAG
GGDRSTARLQCGHEFHLDCIGSAFNAKGVMQCPNCRQIERGNWLYANGSRPSQDVSNDDD
WGHDEDFYDANQPETSRSVFLPFRFQWCPIGRLAQLPSVFDEGESAPPVTFHDFMGQNFT
SEHLPVSAPGATPPGPYIAYFQPLQSSASSSSSHVTERTMDGTTYHDHWNPLPGPSDGRPL
ATVHPIDFHHNHWTHLPNSYSQPNSNNGVAEQMAIPVVPMRVGGLDSDSQQRGSLPSVYGN
GSGSRSRIPSVPPMAPQFMRPHGNINEQYQQNSSSLYAAPQRRTAVQAVQDSMNFTLFPQA
PTGPNSMETEDAGGNQFYAWERDRFAPYPLMPVDSEANWWGSTPQSHGVTDHSAAPGRRLFG
QWIGAGRSPPPPPPPPADNSSYRQMHIPRM

---

Observed pLDDT values

Condition	Source	Loops	Steps	Seeds	pLDDT (mean)
Full + MSA	Biohub API (forge.biohub.ai)	3	50	1	0.812
Full + MSA	Biohub API (forge.biohub.ai)	3	50	1	0.775 (no MSA)
Full + MSA	HuggingFace local (@GregPreibisch)	3	50	5	0.630–0.642
Full + MSA	HuggingFace local (@GregPreibisch)	20	50	5	0.630–0.657
Full, no MSA	HuggingFace local (@GregPreibisch)	3	50	5	0.530–0.569
Full, no MSA	HuggingFace local (@GregPreibisch)	20	50	5	0.506–0.537
Full (HPC)	Biohub HPC (esmc fold --model full)	20	50	1	0.789
Fast, no MSA	Biohub API	3	50	1	0.800

The gap between API and local is ~0.17–0.18 pLDDT units across all conditions. This is not a stochastic seed effect — the local runs used 5 seeds and show tight variance (±0.01–0.02).

---

Reproduction (API side)

from esm.sdk.forge import SequenceStructureForgeInferenceClient
from esm.sdk.api import FoldingConfig, MSA

sequence = (
    "MGAEEEEEPASAVGREGGGGGGGARAAGAGAGGDTADDDDSGESAAAVVPCSICLDAVVA"
    "GGGDRSTARLQCGHEFHLDCIGSAFNAKGVMQCPNCRQIERGNWLYANGSRPSQDVSNDD"
    "DWGHDEDFYDANQPETSRSVFLPFRFQWCPIGRLAQLPSVFDEGESAPPVTFHDFMGQNF"
    "TSEHLPVSAPGATPPGPYIAYFQPLQSSASSSSSHVTERTMDGTTYHDHWNPLPGPSDGRP"
    "LATVHPIDFHHNHWTHLPNSYSQPNSNNGVAEQMAIPVVPMRVGGLDSDSQQRGSLPSVYG"
    "NGSGSRSRIPSVPPMAPQFMRPHGNINEQYQQNSSSLYAAPQRRTAVQAVQDSMNFTLFPQ"
    "APTGPNSMETEDAGGNQFYAWERDRFAPYPLMPVDSEANWWGSTPQSHGVTDHSAAPGRRL"
    "FGQWIGAGRSPPPPPPPPADNSSYRQMHIPRM"
)

client = SequenceStructureForgeInferenceClient(
    url="https://forge.biohub.ai",
    token=YOUR_API_KEY,
    model="esmfold2-2026-05"
)

# With MSA (2048 seqs subsampled from ColabFold MSA)
config = FoldingConfig(num_loops=3, num_sampling_steps=50, include_pae=True)
result = client.fold(sequence=sequence, msa=msa, config=config)
print(f"API pLDDT: {result.plddt.mean():.3f}")  # → 0.812

Reproduction (local HuggingFace side, credit: @GregPreibisch)

from transformers.models.esmfold2.modeling_esmfold2 import ESMFold2Model

model = ESMFold2Model.from_pretrained("biohub/ESMFold2").cuda().eval()
output = model.infer_protein(sequence, num_loops=3, num_sampling_steps=50)
print(f"Local pLDDT: {float(output['plddt'].mean()):.3f}")  # → 0.630–0.642

---

Additional context

• The protein is ~88% intrinsically disordered (AF3 fraction_disordered = 0.88–0.89). The discrepancy is most visible on the IDR — the RING domain (aa 48–107) shows high pLDDT in both API and local runs, but the disordered tail is where the gap is largest.
• We compared against AlphaFold3, Chai-1, and Boltz-2 on the same protein. All three give global pLDDT in the 0.31–0.50 range for this protein, consistent with the local ESMFold2 results — not the API results.
• The Biohub HPC tool (esmc fold --model full, same GPU backend) also returns pLDDT ~0.789, suggesting the discrepancy may be specific to the forge.biohub.ai API endpoint rather than the model weights themselves.
• @GregPreibisch noted on Twitter: "I've seen that too, but only for people who used the web version, not local" — suggesting this may be a known issue affecting the web/API endpoint specifically.
• Tom Sercu (@TomSercu) from the Biohub team has asked for a MWE to investigate.

---

Question

Is there a pLDDT rescaling, temperature adjustment, or post-processing step applied at the forge.biohub.ai API endpoint that is not present in the HuggingFace weights? Or is there a version mismatch between the deployed API model and the released weights?

Materials used
-ALL files zipped here:

github_issue_OsIPI1_ESMFold2_calibration.zip

-Input data

OsIPI1_sequence.fasta — canonical 455 aa sequence
OsIPI1_full_msa_11634seqs.a3m — full MSA (11,634 sequences from ColabFold)
-ESMFold2 API outputs

esmfold2_api_full_msa_structure.pdb — predicted structure (Full + MSA)
esmfold2_api_full_msa_plddt.npy — per-residue pLDDT array
esmfold2_api_full_msa_pae.npy — PAE matrix (455×455)
esmfold2_api_full_msa_confidence.json — raw confidence JSON from API
esmfold2_api_full_nomsa_plddt.npy — Full, no MSA
esmfold2_api_fast_nomsa_plddt.npy — Fast, no MSA
-HuggingFace local outputs (credit: @GregPreibisch)

esmfold2_local_huggingface_GregPreibisch.zip — all 20 PDB structures + Metrics.csv (4 conditions × 5 seeds)
-Cross-model comparison

cross_model_comparison_summary.csv — all 8 predictors in one table
figA_global_ring_plddt_all_models.png — bar chart comparing all models
per_residue_plddt_all_models.png — per-residue pLDDT across all models
esmfold2_msa_effect.png — API no-MSA vs +MSA comparison
structure_visualization_plddt_colored.png — 2×2 PyMOL renders

---

References

• Twitter thread: https://x.com/AyushmanMallick/status/2060780580690907584?s=20
• PeptoneBench (systematic IDP benchmarking): https://github.com/PeptoneLtd/PeptoneBench
• Reference structure: PDB 8I3X (OsIPI1 RING domain, 92% identity)

---

[GrzegorzPreibisch]

@Ayushmania2002 Hey, good work. My hypothesis is that pLDDT is not calibrated well for flexible regions.
I think to actually benchmark this well I would suggest aggregating several extremly similar PDB files with flexible regions. For example a good case would be RSV site 0 pre-fusion state - different structures differ from each other in this region around 0-1.5 RMSD-ca. LDDT of this region should be higher bound for pLDDT value.
Do you have a good example from your domain?

[Ayushmania2002]

@GrzegorzPreibisch Thanks for the detailed suggestion! The RSV site 0 example is a great benchmark design, using structural ensemble LDDT as the pLDDT upper bound is exactly the right framing.

From our domain, I think PYL/RCAR ABA receptors would be an ideal test case. The PYL family has ~50+ crystal structures in the PDB across 14 paralogs, and the key flexible regions are well-characterized: the gate loop (CL1) and latch loop (CL2) undergo a defined open→closed conformational change upon ABA binding
Nature

Science
. Crucially, multiple structures of the same paralog exist in both apo and ABA-bound states, for example PYL1 has structures in open (apo) and closed (ABA-bound) conformations, and the gate/latch loops differ by ~1–3 Å RMSD between crystal forms while the β-barrel core is essentially rigid
Nature

Nature Structural &Molecular Biology.

The gate and latch loops are exactly the regions where ESMFold2 API reports inflated pLDDT (~0.81) while local HuggingFace weights give ~0.63, and where Boltz-2 is most honest (~0.31). A benchmark using e.g. PYL1 (PDB: 3K3K, 3KDI, 3UQH) or PYL2 (3KDH, 3KAZ) would let you compute the true inter-structure LDDT of the gate/latch loops and directly compare it to what each predictor reports as pLDDT for those residues.

The rice OsPYL/RCAR structures (e.g. OsPYL3, PDB: 4LG5) add a crop-species angle. Will need to pull the relevant PDB IDs and compute the inter-structure LDDT, when I manage time this week.

[GrzegorzPreibisch]

@Ayushmania2002 I've posted a quick analysis.
https://x.com/GregPreibisch/status/2061532108573622741

Am I understanding right that I should generalize this analysis to 2 groups (3K3K, 3KDI, 3UQH) and (3KDH, 3KAZ)? I need at least 2 similar reference PDB to calibrate LDDT baseline. Hit me up if I understand correctly that those analogues are fitting into this framework of analysis.

[Ayushmania2002]

@GrzegorzPreibisch yes, you've done it correctly, saw the post reply.