ParliaBench — UK Parliamentary Speech Generation

About ParliaBench

ParliaBench is a benchmark and evaluation framework for LLM-generated UK parliamentary speeches, combining a curated dataset, multi-dimensional evaluation metrics, and five domain-specific fine-tuned models.
Paper: arXiv:2511.08247

Constructed from the UK subset of the ParlaMint corpus, 2015–2022. Four-step pipeline: XML parsing → metadata alignment → content filtering → EuroVoc thematic classification.

Five LLMs fine-tuned with QLoRA via the Unsloth framework:

System prompt — training (no word count):

You are a seasoned UK parliamentary member. Use proper British parliamentary language
appropriate for the specified House. The speech should reflect the political orientation
and typical positions of the specified party on the given topic.

System prompt — generation (includes word count target):

You are a seasoned UK parliamentary member. Generate a coherent speech of
{min_words}-{max_words} words in standard English (no Unicode artifacts, no special
characters). Use proper British parliamentary language appropriate for the specified
House. The speech should reflect the political orientation and typical positions of the
specified party on the given topic.

Context string (pipe-separated, injected as user turn):

EUROVOC TOPIC: {topic} | SECTION: {section} | PARTY: {party} | POLITICAL ORIENTATION: {orientation} | HOUSE: {house}

Model-specific chat templates

<s>[INST] {SYSTEM_PROMPT}
Context: {context}
Instruction: {instruction} [/INST] {response}</s>

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
{SYSTEM_PROMPT}<|eot_id|><|start_header_id|>user<|end_header_id|>
Context: {context}
Instruction: {instruction}<|eot_id|><|start_header_id|>assistant<|end_header_id|>
{response}<|eot_id|>

<bos><start_of_turn>user
{SYSTEM_PROMPT}
Context: {context}
Instruction: {instruction}<end_of_turn>
<start_of_turn>model
{response}<end_of_turn>

<|im_start|>system
{SYSTEM_PROMPT}<|im_end|>
<|im_start|>user
Context: {context}
Instruction: {instruction}<|im_end|>
<|im_start|>assistant
{response}<|im_end|>

Every generated speech passes a 9-step validation pipeline. Invalid speeches are automatically regenerated up to 3 times. Baseline models exhibited higher failure rates, suggesting fine-tuning improved output quality directly.

green = significant improvement   red = significant regression   PPL ↓ · Self-BLEU ↓ · all others ↑

↑/↓ p < 0.05 after Bonferroni correction. PSA and Party Align on 0–1 scale; J scores on 0–10 scale; GRUEN/BERT/Mover on 0–1; PPL raw.

These are the two novel embedding-based metrics introduced by ParliaBench to measure political authenticity — dimensions entirely unavailable to conventional NLP metrics.