Political provocation classifiers and the Toxicity Index (TI7)
Eight SetFit classifiers for tactics of political provocation in short social media posts, released with O'Neill, Cabanillas, Kepner, Goggin, and Brooks (2026), The Toxicity Index: measuring political provocation in the United States Congress (manuscript in preparation). Three of them, together with four general-purpose classifiers, produce the seven-component Toxicity Index (TI7) used in the paper.
Classifiers in this repository
| Folder | Construct | Positive class | Role in the paper |
|---|---|---|---|
setfit_ragebait |
Ragebait | Content framed to provoke outrage beyond its informational content | TI7 component |
setfit_tribal_signal |
Tribal signaling | Us-versus-them framing marking in-group loyalty or out-group hostility | TI7 component |
setfit_performative_outrage |
Performative outrage | Theatrical indignation staged for an audience | TI7 component |
setfit_political_advocacy |
Policy advocacy | Substantive communication of policy positions or legislative action | Reported beside TI7 |
setfit_genuine_anger |
Genuine anger | Text coded as angry rather than staged (see the caution below) | Excluded from TI7 |
setfit_epistemic_manipulation |
Epistemic manipulation | Cherry-picking, straw-manning, false equivalence | Released, not analyzed |
setfit_engagement_bait |
Engagement baiting | Polls, calls to action, rhetorical bait | Released, not analyzed |
setfit_agency_language |
Agency language | Active, agentic framing (versus passive or victim framing) | Released, not analyzed |
Each classifier turns a post into a numerical representation with a fine-tuned sentence-embedding model (sentence-transformers all-mpnet-base-v2) and applies a logistic-regression head (SetFit). Load with SetFitModel.from_pretrained("jimnoneill/xbox-compulsion-classifier", subfolder="setfit_ragebait") and use predict_proba for the positive-class probability.
Training data
Labels were produced in two steps. A language model assigned initial scores from 0 to 1 under written definitions with examples from both parties, and one author then reviewed every score and corrected it where needed.
| Pool | Posts | Source | Dates | Classifiers |
|---|---|---|---|---|
| A | 1,972 | One senator's personal account (X API archive) | 25 Jul 2022 to 22 Feb 2026 | Ragebait and epistemic manipulation (pool A only); the four below with pool B |
| B | 1,949 | 13 senators' accounts, from the Hugging Face dataset m-newhauser/senator-tweets | 20 Jan to 30 Dec 2021 | Tribal signaling, performative outrage, engagement baiting, agency language (with pool A) |
| B | 200 | Donald Trump's account, from the Trump Twitter Archive (the one noncongressional account) | 2010 to 2017 | As above |
| C | 1,000 | 562 congressional accounts in the congresstweets archive | 2017 to 2023 | Policy advocacy, genuine anger |
The 14 accounts of pool B are balanced by party: seven Democratic (counting Senator Sanders) and seven Republican.
Reviewed scores were converted to present and absent labels with construct-specific cutoffs; posts between the cutoffs were left out.
| Classifiers | Present at or above | Absent at or below |
|---|---|---|
| Ragebait, epistemic manipulation | 0.65 | 0.35 |
| Tribal signaling, performative outrage, engagement baiting, agency language, policy advocacy | 0.65 | 0.30 |
| Genuine anger (clearly angry posts were rare) | 0.40 | 0.10 |
Performance
Five-fold cross-validation against the reviewed labels. Posts were grouped by account where account identifiers existed and by exact text for pool A, so that the same account or text never contributes to both sides of a fold. Each fold retrains with the original hyperparameters and training-set size.
| Classifier | n | AUC (SD) | F1 at 0.5 (SD) | Precision | Recall |
|---|---|---|---|---|---|
| Ragebait | 1,851 | 0.899 (0.020) | 0.795 (0.017) | 0.741 | 0.860 |
| Tribal signaling | 3,336 | 0.905 (0.027) | 0.737 (0.047) | 0.659 | 0.848 |
| Performative outrage | 3,836 | 0.907 (0.023) | 0.751 (0.038) | 0.679 | 0.847 |
| Epistemic manipulation | 1,420 | 0.931 (0.022) | 0.823 (0.031) | 0.786 | 0.864 |
| Engagement baiting | 4,092 | 0.883 (0.036) | 0.648 (0.056) | 0.550 | 0.791 |
| Agency language | 3,151 | 0.883 (0.034) | 0.748 (0.038) | 0.702 | 0.807 |
| Policy advocacy | 995 | 0.931 (0.013) | 0.905 (0.021) | 0.924 | 0.888 |
| Genuine anger | 901 | 0.970 (0.021) | 0.766 (0.082) | 0.684 | 0.887 |
Cross-validation measures agreement with the training-label scheme. It is not independent validation and does not show that the probabilities are calibrated.
Human-coding pilot: 150 congressional posts were drawn from four strata defined by classifier scores (60 from the optimism class of the general-purpose emotion model, 30 with high advocacy scores from its other classes, 30 with low advocacy scores, and 30 random posts, balanced by party). The first coder affirmed a sheet prefilled with model scores; the second coder independently completed a blank sheet. Krippendorff's alpha was about 0.83 (genuine anger), 0.80 (advocacy) and 0.81 (performative outrage). Against the second coder's labels, the performative-outrage classifier reached AUC 0.93 and the policy-advocacy classifier 0.75. The pilot is a diagnostic check of three constructs, not a population performance estimate.
Caution on the genuine-anger classifier
Against the second coder, the genuine-anger classifier ranked human-coded anger well (AUC 0.87), but only ten of the 150 pilot posts were anger-positive. Across the 5,072,129 scored records of the congressional corpus it flagged 29 percent of posts, and its score correlated strongly with performative outrage (Spearman rho 0.79). It behaves as a hostility and partisan-attack detector and is excluded from TI7. Use it only with that interpretation.
The anger class of the general-purpose TweetNLP emotion model (cardiffnlp/twitter-roberta-base-emotion) behaves in the same way. Its anger probability ranked the second coder's anger labels with AUC 0.93, it assigned 29 percent of the scored records to anger, 77 percent of those posts also scored above 0.5 on the genuine-anger classifier, and its anger probability correlated with performative outrage at rho 0.66. Both scores are excluded from TI7, because a score that marks nearly a third of congressional posts and moves this closely with staged indignation does not isolate anger.
Class order of the emotion model (correction of 15 November 2026)
The outputs of cardiffnlp/twitter-roberta-base-emotion are in the order of the TweetEval mapping file: anger, joy, optimism, sadness. The id2label table in that model's config.json lists another order (joy, optimism, anger, sadness) and must not be used to name the outputs. The xbox package in this repository fixes the order in xbox/config.py (CLASS_ORDER_OVERRIDES), and the analysis repository tests it on sentences of known emotion.
Before this date the package and the analysis code read the order from config.json, so what they reported as the emotion model's anger score was its optimism probability. Two statements on earlier versions of this card described that optimism score and are withdrawn: that the emotion model's anger score ranked the coder's anger labels inversely (AUC 0.13), and that most pilot posts it labeled anger were coded as policy advocacy. TI7 and its seven components do not use the emotion model and are unchanged.
Toxicity Index (TI7)
TI7 for a post is the mean of seven positive-class probabilities: offensive language, toxicity, negative sentiment and irony (general-purpose), and ragebait, tribal signaling and performative outrage (this repository). It ranges from 0 to 1. A member's or account's score is the mean over non-repost posts. Policy advocacy is reported as a separate score.
| General-purpose classifier | Model |
|---|---|
| Sentiment | cardiffnlp/twitter-roberta-base-sentiment-latest |
| Emotion | cardiffnlp/twitter-roberta-base-emotion |
| Offensive | cardiffnlp/twitter-roberta-base-offensive |
| Irony | cardiffnlp/twitter-roberta-base-irony |
| Hate | cardiffnlp/twitter-roberta-base-hate-multiclass-latest |
| Toxicity | s-nlp/roberta_toxicity_classifier |
Code and archives
- Analysis code: https://github.com/jimnoneill/xbox-toxicity-index
- Training labels (with post text) and annotation: https://huggingface.co/datasets/jimnoneill/xbox-political-compulsion-dataset
- Software archive: https://doi.org/10.5281/zenodo.24144555 (version 2026.11.11)
- Derived results and annotation metadata, including the numeric training-label projections and per-classifier cutoffs: https://doi.org/10.5281/zenodo.24144556 (version 2026.11.11)
The xbox/ package in this repository contains the scoring code and the analysis modules (stream hygiene, rhythm descriptors, roll-call location model, point-process models, binned count models and the forecasting experiments).
Superseded files
bayesian_model_results.json, calibrated_model_v2.json, cohort_signatures.csv, cohort_v2_results.csv, and the modules xbox/compulsion_signatures.py, xbox/disorder_baseline.py and xbox/fusion.py belong to an earlier version of this project that estimated a compulsion probability from posting rhythm. That model is not used in the paper and its earlier validation figures should not be relied on.
Intended use and limits
Research on political communication at the population level. The scores describe text; they are not designed to support inferences about the mental health or character of any individual. Training labels were reviewed by a single author, and only performative outrage among the TI7 components was checked in the human-coding pilot. TI7 is an operational composite that has not been independently validated as a scale.
Citation
O'Neill, J., Cabanillas, J., Kepner, W., Goggin, S. N., & Brooks, J. (2026). The Toxicity Index: measuring political provocation in the United States Congress. Manuscript in preparation.