How I built an analysis engine for Italian parliamentary speech

See the website here. Note that I am still working on the frontend, so some of the analytics are not yet available.

Every time I watch a parliamentary debate I end up asking the same things. Does this person only ever talk about the same three subjects, or do they follow whichever way the wind blows? Is their group actually united, or are there five different positions inside it? How far is their language really from the opposition's?

None of that is answerable from a single speech. You need thousands of them, and you need something that will read them for you.

The project is called Parliament Speech Analyzer (creativity, clearly, at its peak) and that is what it does: it takes the official list of Assembly sittings, downloads the stenographic reports, turns them into vectors, assigns them to topics, computes about ninety metrics, and emits a JSON payload that a React frontend turns into maps and rankings.

This article is about how it works inside. The architectural decisions, the algorithms, and mostly the places where the domain is more hostile than it looks.

The data exists. It's public, too. It is not convenient.

Both chambers publish the full stenographic record of every sitting. Every word spoken in the chamber is there, free, no registration. The problem is that these are documents built to be read by a human interested in one particular day, not processed by a machine interested in fifteen months.

The distinction that holds up the whole ingestion layer is this: which sittings exist and what was said in them are two different questions, and they go to two different sources.

The first one has an official, structured answer. Both chambers publish their calendars and their member registers as linked data, with open SPARQL endpoints:

PREFIX osr: <http://dati.senato.it/osr/>
SELECT ?seduta ?data ?numero WHERE {
  ?seduta a osr:SedutaAssemblea ;
          osr:legislatura 19 ;
          osr:dataSeduta ?data ;
          osr:numeroSeduta ?numero .
  FILTER (?data >= "2025-05-01"^^xsd:date)
}

dati.camera.it and dati.senato.it give me the complete list of sittings for the 19th legislature and the member registers. Only the speech text itself comes from scraping the chambers' websites, because that part isn't in the open data.

Linked data is wonderful until you actually use it. Three things that cost me an afternoon each, now written into the comments of backend/ingestion/sparql.py so nobody has to rediscover them: dati.senato.it answers 403 unless you send a browser-ish User-Agent; osr:legislatura is an xsd:integer, so filtering on the string "19" matches nothing and complains about nothing; and on the Camera side, seduta.rdf/s19_<n> URIs are Assembly sittings while BF_19_* ones are bulletins, so including them leaves you with a corpus full of things nobody ever said out loud.

Starting from the official list buys one thing that outweighs everything else: I know how many sittings there should be. Which means I can measure how many I actually got.

The architecture

The backend runs locally or on Colab, produces static JSON, and the frontend consumes it with no API in between. That's deliberate: parliamentary records don't change in real time, so there is nothing to serve dynamically. The frontend sits on any CDN, there's no endpoint to version, and testing a UI edge case only takes a hand-written fake JSON file.

The crawler, or: the part that has to know how to say no

Everything that isn't chamber-specific parsing lives in one place: concurrency, rate limiting, per-sitting caching, coverage accounting. Both chambers get it for free.

Two properties I wanted explicitly.

It resumes. Parsed speeches are cached per sitting, not at the end of the run. A run that dies at sitting 400 of 443 costs 43 fetches to finish, not 443. The crawler's first pass is always over the cache, so a resumed run never touches the network for something it already has.

It's accountable. At the end of a run, CrawlReport says: how many sittings the open data says exist, how many were fetched, how many came from cache, how many parsed successfully, how many speeches came out, how many were blocked, how many failed, and the coverage percentage.

That matters because the interesting failure in this domain isn't an exception, it's a polite answer. senato.it sits behind a CloudFront JavaScript challenge: to an automated client it replies HTTP 202 with two kilobytes of interstitial explaining that we need to verify you're not a robot. It's a 2xx response containing valid HTML. A naive parser receives it, looks for speeches, finds none, and calmly returns an empty list.

So challenges are detected and raised as ChallengeBlocked, never swallowed:

CHALLENGE_MARKERS = (
    'challenge-container',
    "verify that you're not a robot",
    'JavaScript is disabled',
    'Checking your browser',
)
CHALLENGE_MAX_BYTES = 8_000   # a real stenographic report is tens of KB

On top of that sits a ResilientTransport that tries plain HTTP first and only falls through to a headless browser (Playwright, an optional dependency) when it's refused. The common case stays cheap and the expensive path engages precisely when it's needed. If Playwright isn't installed the run doesn't break: it reports how many sittings stayed out, and you know why the Senate has less data than the Chamber.

There's a command whose only job is this, meant to run before a real export:

python -m backend.ingestion.verify --source both --months 15

It prints per-chamber coverage, the date range, how many speakers were resolved ambiguously and by which strategy. It exits non-zero when a chamber is blocked, so it works as a preflight check in a script. It's the most boring thing I've written in this project and the one that saves me the most time.

Who spoke

Stenographic reports name members however they feel like: MALAN, Giuseppe Conte, CONTE Giuseppe. The official register stores Surname FirstName. In between sits a matcher that indexes the register once (exact, caseless, by surname) and tries the forms in order of reliability.

The part that matters is what it does when it isn't sure. If two members share a surname it tries to disambiguate by parliamentary group; if that fails it takes the first and marks the attribution as ambiguous. That flag travels all the way to the run report. A doubtful attribution is data, not a detail to hide behind a [0].

And when a name isn't in the register at all, the matcher returns None. Which is a useful result: it rejects parser false positives, like a capitalised word mid-speech being read as a speaker.

The NLP pipeline

From raw text to payload, in order:

Cleaning — procedural formulas stripped, text normalised. Presidency interventions are excluded, along with anything under 30 words.
Embedding — each speech becomes a 384-dimension vector via paraphrase-multilingual-MiniLM-L12-v2.
Reduction — PCA down to two dimensions for the map.
Topic assignment — 14 predefined areas, cosine similarity, with a floor.
Centroids — one mean vector per topic, which the distance metrics need.
Nine analyzers — over the whole corpus and over every period that can support the analysis.
Export — a manifest plus chunks under frontend/public/data/.

Why sentence embeddings and not TF-IDF

TF-IDF counts words. Two speeches about the economy that use different vocabulary come out far apart even when they're saying the same thing. Embeddings encode meaning, so they land near each other despite different lexicons. In a corpus where every group has its own dialect for discussing the exact same subjects, that difference isn't academic.

I picked MiniLM-L12 for three concrete reasons: it handles Italian natively with no translation step, 384 dimensions is a reasonable trade, and it's small enough to run on CPU in times that don't make iteration unbearable. Something like multilingual-e5-large would give better results and a pipeline I'd never feel like launching.

PCA rather than t-SNE

I implemented both; the default is PCA. t-SNE produces visually cleaner, better-separated clusters and much nicer screenshots, but distances between clusters in the 2D plane mean nothing: it preserves local structure and distorts global structure. With PCA the clusters are blurrier, but if two points are far apart on the chart they really are far apart semantically. On a map people will use to draw conclusions, the second property beats the aesthetics.

Fourteen topics and the right not to answer

Classic K-Means is still there as a fallback (12 clusters, chosen with the elbow method back when), but the default is semantic assignment: fourteen predefined areas — tax and public finance, labour and business, health, welfare and family, environment and energy, justice, immigration, civil rights, education, agriculture, foreign affairs and defence, infrastructure and transport, constitutional and regional reform, electoral reform — each described by a keyword list, each embedded, and every speech goes to the nearest one.

The upside is interpretability: topics carry the same meaning from one run to the next, which is not true of K-Means. The cost is that someone (me) decided up front what the topics of Italian politics are, which is a bias and should be stated as one.

The technically interesting problem is that argmax always answers. It has no way to say "this intervention isn't about any of these things." And a stenographic record contains a lot of material that is about none of these things: points of order, housekeeping announcements, thanking the chair.

So there's a floor. Below 0.20 cosine similarity the speech stays -1, "unclassified", instead of being filed under the least-bad match. And every assignment carries a confidence margin:

partitioned = np.partition(similarities, -2, axis=1)
confidence = partitioned[:, -1] - partitioned[:, -2]   # best minus runner-up

A small margin means the speech matched two topics about equally well and the choice was effectively a coin flip. The payload says so, and the frontend shows it.

One limitation that's written into the methodology note rather than hidden: the resulting distribution is skewed, because institutional and procedural language gets pulled toward a handful of areas. Comparisons between topics have to be read with that in mind.

A type that makes one particular bug impossible

Nearly every metric in this project performs the same manoeuvre: mask a DataFrame, then use the result to slice a numpy array of the same length.

mask = df['group'] == 'Fratelli d\'Italia'
party_df = df[mask]
party_emb = embeddings[mask]

That works as long as the DataFrame carries a clean 0..n-1 index. One sort_values('date'), one concatenation, one forgotten reset_index, and pandas keeps indexing by label while numpy indexes by position. Rows and vectors stop corresponding.

And nothing happens. No exception, no warning, no shape mismatch: the means still compute, the cosine similarities still compute. The numbers simply become somebody else's. In a project whose end product is a number, that is the worst failure mode available.

So the frame and its arrays never travel apart:

@dataclass(frozen=True)
class SpeechDataset:
    df: pd.DataFrame
    embeddings: Optional[np.ndarray] = None
    topic_scores: Optional[np.ndarray] = None

There's exactly one way to narrow it, subset(), which converts any boolean mask to positions before using it and rebuilds the index at every step. The constructor checks that the lengths agree. There is no API through which a misaligned slice can be expressed.

ds = SpeechDataset(df, embeddings=emb, topic_scores=scores)
camera = ds.subset(ds.df['source'] == 'camera')
for month, bucket in ds.by_period('month'):
    ...

The invariant isn't documented, it's structural. That's the only version of it I trust to survive a refactor six months from now by someone who doesn't remember why it's there.

Caching, which here is a requirement rather than an optimisation

Embedding eight thousand speeches on CPU takes minutes. Redoing it every time you touch a threshold means never touching a threshold again.

The store is single and content-addressed: keys are kind / source / digest, where the digest describes what the artifact was computed from.

Scraped speeches are keyed on the fetch parameters (legislature, months back), because you can't know the content before fetching it.
Embeddings are keyed on a SHA-256 of the exact texts they encode, plus the model name.

That second line is the whole point. A corpus can change while keeping the same row count — one re-scrape that catches different sittings will do it — and a cache validated on the count would cheerfully reuse vectors belonging to other texts, making every downstream metric wrong without telling anyone. With a content fingerprint, one different character means a different key, so the cache simply misses. There is deliberately no fallback to an unfingerprinted file.

Since every content change writes a new file, there's a prune() that keeps the two newest entries per kind and source. With one detail I learned the traditional way: several writes can land inside the same filesystem timestamp tick, and with equal mtimes the sort order is arbitrary. Without a protect parameter, a run can delete the embeddings it just finished computing.

The nine analyzers

An abstract BaseAnalyzer with a compute() method, and each analyzer registers itself through a decorator. The orchestrator discovers them from the registry, injects the shared data, and runs them.

@analyzer
class AlliancesAnalyzer(BaseAnalyzer):
    name = "alliances"
    min_speeches = 200        # cross-party mixing needs volume to be visible
 
    @classmethod
    def get_dependencies(cls) -> list[str]:
        return ['embeddings']

Briefly, what they compute:

Identity — the thematic DNA of each group and member: how much room they give each area, lexical richness, distinctive terms via TF-IDF. The generalism index is the entropy of the topic distribution: someone who only talks about tax has low entropy, someone who touches everything has high entropy.

Sentiment — tone per topic, a group × topic matrix, rankings, the Gulpease index (readability calibrated for Italian: below 55 the text is hard going for a reader with lower-secondary education) and "us versus them" polarisation. All of it lexical, meaning it counts marker words. A transformer model can be switched on in config for anyone with time to spare, but the default stays the count, which runs in milliseconds and can be explained to a person.

Temporal — how topics move month to month, semantic drift of each group through embedding space, a crisis index counting alarm terminology, and topic surfing, meaning abrupt shifts of thematic focus.

Relations — the affinity matrix between groups, still my favourite output: mean cosine similarity across all of their speeches. Every so often it turns out that two parties who savage each other on television have almost interchangeable language. Internal cohesion measures the opposite: how tightly a single group's speeches sit together.

Speaker — verbosity, rhetorical questions, self-reference, regularity of interventions, named entities, topic leadership.

Rhetoric — populism (the people versus the elite), anti-establishment (system, caste, palace), emotional intensifiers, institutional register. Word counting normalised by length. It isn't sophisticated NLP, but for this kind of pattern it works surprisingly well and it's infinitely more interpretable than a black box.

Factions — a conformity score against the group's own centroid, with mainstream, bridge (someone sitting halfway between their group and another) and maverick labels.

Alliances — cross-cutting topics, unusual pairings across the aisle, left-right alignments on individual subjects.

Topics — cluster labels and keywords, with a POS filter that keeps only nouns and adjectives.

The part I find most interesting: when an analyzer declines

The frontend has a period selector, so analytics are also computed per year and per month. But not every metric survives that cut.

Each analyzer declares the conditions under which its numbers mean something:

period_safe: bool = True    # False when it measures change *between* periods
min_speeches: int = 30      # smallest sample worth publishing

temporal sets period_safe = False, because asking for the semantic drift of a single month is asking how much something changed inside a photograph. alliances wants 200 speeches, because cross-party mixing isn't observable in a sample where half the groups appear three times. The orchestrator asks, the analyzer answers with a reason, and the reason ends up in the report.

Because there is a report. An analyzer that blows up gets recorded rather than aborting the export — one broken metric shouldn't cost the whole run — but the failure has to surface somewhere, or the payload ships with a hole in it that nobody notices. AnalyticsRunReport tracks what failed and in which period, the list lands in stats.analytics_run, and --strict makes the pipeline exit non-zero when something went wrong. A period where every analyzer declined doesn't get a file at all: the frontend notices, falls back to the global block, and says so.

A metric with no unit is an opinion with decimal places

Several metrics here count marker words and then have to turn that count into a number printed next to a person's name. How you do that matters.

There's one shape, defined in backend/scoring/normalize.py, and every lexicon metric uses it:

raw — markers per thousand words. A unit you can state out loud, comparable across corpora, with no ceiling.
pct — percentile rank within the corpus. This is what drives bar lengths and ranking order, because "more than 90% of their colleagues" is a claim the data supports, whereas "82 out of 100" means nothing.
n — how many speeches the value rests on, so a thin sample is visible instead of camouflaged.

The "low / medium / high" labels refer to the percentile, not to absolute cutoffs. A score with a ceiling is worse than useless: it flattens exactly the tail where the measure claims to be most informative.

Same philosophy on naming. There's a metric for the share of a member's interventions falling outside their group's dominant topic area. It's called divergence_pct, "thematic independence" in the frontend, and the methodology note specifies that it measures distance from the group's agenda and not political dissent: someone sitting on a different committee from most of their colleagues scores high without ever having contradicted anyone. Calling it a "rebel score" would have been much more fun and much more false.

The export: a manifest and some pieces

The payload isn't one file per chamber. It's an index plus resources the frontend fetches when it needs them:

data/
  manifest.json                  what exists, how big, which periods
  camera/core.json               deputies, clusters, stats — first paint
  camera/speeches.json           only the map asks for this
  camera/analytics/global.json
  camera/analytics/2025.json
  camera/analytics/2025-11.json

Numbers from the latest Camera run: 8,240 interventions by 270 deputies across twelve groups, 115 of which stay unclassified, with fourteen months and two years of separate analytics. core.json weighs 774 KB and is everything needed to draw the interface. speeches.json weighs 8.4 MB and only downloads for people who open the map.

ArtifactWriter writes each resource, computes its digest, records it in the manifest, and checks declared size budgets: half a megabyte for the manifest, 3 MB for core, 12 for speeches. Going over doesn't block the run, it prints a warning. It's the kind of regression you'd otherwise find six months later, when someone on a normal connection tries to open the site.

No indentation either, incidentally. indent=2 on a payload this size costs megabytes of whitespace that no human being will ever read.

A side effect I'm happy about: chunks are cheap to keep in git. A new run only rewrites the periods that actually changed; the rest stay byte-identical and git stores them once.

The frontend

React with Vite, react-router, Plotly for the map, Tailwind. The routes are in Italian — /mappa, /analisi/identita, /analisi/relazioni, /analisi/tendenze, /analisi/qualita, /analisi/parlamentari — and the chamber is a query param, so any view is a link you can send to someone.

Loading follows the shape of the payload: manifest and core.json on first paint, speeches and the period's analytics on the first view that genuinely asks for them, everything held in a session cache so switching chamber or month never refetches what's already there. Anything data-dependent sits under a ChamberBoundary that owns the loading and error states, so six pages don't rewrite the same logic six times.

Two visualisation decisions I made against my own initial instinct.

Chart series colours are not party colours. Institutional group colours don't pass distinguishability checks: Lega's green and PD's red are nearly identical to someone with a colour vision deficiency, and those two parties end up in the same chart essentially always. Party colours are still there, but as a swatch next to a text label. Series use a verified scale with a limited number of elements. And the map doesn't colour fourteen topics at once: it highlights at most three things at a time and keeps the rest as neutral context.

There's a /metodo page. One document explaining how each metric is computed and what it doesn't measure: that the two chambers aren't comparable because they cover different time windows, that sentiment is lexical and doesn't catch irony, that affinity between groups indicates overlapping language rather than political closeness, that an empty panel usually means "below threshold" and not "error". And that speaking isn't legislating: nothing here measures votes, attendance, or legislative outcomes.

A project that produces numbers about politics and doesn't state its own limits is doing something worse than being wrong.

The tests

219 functions across 22 files. Not a number to boast about, since tests count badly. What I care about is what they check.

The most useful ones are the invariant tests, and the technique is simple: build arrays where row i carries the value i, then check after every slice that the array's contents still match the DataFrame's row_id column. If the slicing is off by a single row, the test shouts instead of producing slightly different numbers. Dates in the fixtures are deliberately out of order, because a sort_values is exactly the scenario that breaks label-versus-position alignment.

Same category: cached embeddings can't be reused for different texts even at identical row counts, a blocked fetch can't become an empty result, and the exported payload matches its contract — which is generated from the code by python -m backend.tools.dump_schema, not hand-written in a markdown file destined to diverge within three weeks.

Running it

python -m venv venv
venv\Scripts\activate
pip install -r backend/requirements.txt
python -m spacy download it_core_news_sm
 
# coverage first, then the pipeline
python -m backend.ingestion.verify --source both --months 15
python -m backend.export_data
 
# Camera only, exit non-zero if an analyzer breaks
python -m backend.export_data --source camera --strict
 
# for when senato.it is being difficult
pip install playwright && playwright install chromium
 
cd frontend && npm install && npm run dev

There's also a Colab notebook for anyone who wants the GPU: it clones the repo, installs, runs, and downloads the payload.

What's left

The code is arranged so that adding a new source — Italy's regional councils publish their records too — means writing one class implementing list_sessions and fetch_session, and nothing else. Concurrency, caching, rate limiting, accounting and nine analyzers come along for free. That was the main architectural requirement from the start: keep the data problem cleanly separated from the analysis problem.

Three things stay open. The topic distribution is still skewed toward a few areas, because procedural language is semantically attracted to them. The Senate depends on a headless browser for as long as CloudFront feels this way about it, so its coverage is far lower than the Chamber's. And some visualisations I have in my head don't exist yet.

But the system now knows how to say what it doesn't know. If a collection is incomplete it prints that in figures; if a metric can't support the sample it skips it and states why; if an attribution is ambiguous it marks it. On a hobby project analysing political speech, that's the one property I treat as non-negotiable.