Scottish Parliament Visualisation

I’ve spent a little time in the last 24 hours (with much help from John Williamson) scraping MSP division (vote) data from the Scottish Parliament website. Divisions are held when the MPs do not agree on a particular motion being put before them. Each MP votes for or against the motion, and the majority wins. The division data is held within the official parliamentary records (essentially a dump of everything said during the day). Here’s an example containing several divisions:

If you know any html, take a look at the source, and marvel at how poorly structured it is (how about defining a div or span class for a division?? Or, as a start, how about closing your divs??). Also note the opaque numbering system (change the number after ‘r=’ in the link to get other records, seemingly randomly ordered).

Ranting aside, I think we finally managed to extract all division data for the current parliament (since 5th May 2011). Assuming I haven’t messed it all up (I will check…), since 5th May 2011 133 different MPs have voted in 580 divisions casting a total of 66,310 votes. I’m happy to share the data or the Python used to scrape it if anyone else wants it.

The first thing I’ve done with the data is visualise the MPs in two dimensions (apologies for the rubbish plot, clicking on it makes it slightly better, labelling the MPs and votes would probably be useful):


In this 2D world, each MP is a point (coloured according to standard party convention, grey are independents), and each vote is a line that splits the MPs who voted ‘for’ the motion from those who voted ‘against’. Given the large number of MPs and votes, it’s incredibly unlikely that it will be possible to position all of the MPs and votes such that all MPs are on the correct side of all votes. However, we can use some computational magic to position them in a manner that gets as many right as possible (in this case, 97.8% of the MP-vote pairs are correct). Once they have been placed, the position of the MPs tells us something about how they vote. For example, they will be near other MPs who have similar voting patterns, and far away from those that have very different ones.

I haven’t had the chance to look at this in much detail yet but the one thing that stands out is the homogeneity of the main parties (their members are very bunched together). In the Westminster parliament, a substantial minority of MPs regularly rebel against their parties resulting in much broader clouds of points (e.g. here is a plot of Westminster MPs since 2010: 2010plain and in the 2005 parliament: 2005plain). The most rebellious MP in the Scottish Parliament is Christine Grahame (SNP) with 6 rebellions (defined as not voting with the party majority) out of 544 votes (~1%). Compare that with Labour MP Dennis Skinner who, up to the end of the 2005 Westminster Parliament had rebelled a total of 273 times (source). In fact, there are so few that I can give you a list of all those MPs who have rebelled more than once:

  • Grahame, Christine (SNP, Midlothian South, Tweeddale and Lauderdale) 6 (544)
  • Murray, Elaine (Lab, Dumfriesshire) 5 (529)
  • Gibson, Kenneth (SNP, Cunninghame North) 4 (569)
  • Malik, Hanzala (Lab, Glasgow) 4 (521)
  • Boyack, Sarah (Lab, Lothian) 3 (523)
  • Chisholm, Malcolm (Lab, Edinburgh Northern and Leith) 3 (569)
  • Cunningham, Roseanna (SNP, Perthshire South and Kinross-shire) 3 (536)
  • Henry, Hugh (Lab, Renfrewshire South) 3 (486)
  • Smith, Drew (Lab, Glasgow) 3 (517)
  • Wilson, John (SNP, Central Scotland) 3 (558)
  • McGrigor, Jamie (Con, Highlands and Islands) 3 (535)
  • Dugdale, Kezia (Lab, Lothian) 2 (545)
  • Eadie, Jim (SNP, Edinburgh Southern) 2 (564)
  • Ferguson, Patricia (Lab, Glasgow Maryhill and Springburn) 2 (492)
  • McLeod, Fiona (SNP, Strathkelvin and Bearsden) 2 (530)
  • Park, John (Lab, Mid Scotland and Fife) 2 (287)
  • Urquhart, Jean (SNP, Highlands and Islands) 2 (274)
  • Carlaw, Jackson (Con, West Scotland) 2 (484)
  • Scanlon, Mary (Con, Highlands and Islands) 2 (553)
  • Fraser, Murdo (Con, Mid Scotland and Fife) 2 (511)
  • Milne, Nanette (Con, North East Scotland) 2 (536)

I’ll leave it with you to decide whether that’s a good or bad thing.

(Note 1: my definition of rebellion is not great – in Westminster, a vote is only rebellious if it is whipped (i.e. the party declare that all MPs have to vote a certain way) and not all votes are. Some of those above may have been on un-whipped votes.)

Two MPs appear on the plot above twice – John Finnie and Jean Urquhart. They both resigned from the SNP in late October 2012 and so appear once for their votes as members of the SNP and once as Independent MPs. In fact, their independent incarnations can be seen as the two grey circles SouthWest of the main chunk of SNP MSPs (yellow).

A final observation is how much voting is done. In the Westminster parliament attendance at votes is highly variable. In Holyrood it seems much more consistent. The median number of votes case by MPs is 530 (out of 580; 91%) and the 25th and 75th percentiles are 492 (85%) and 558 votes (96%) respectively.

Note 2: An added complication with the Scottish Parliament is that MPs can actively abstain from a vote. The model deals with this by defining an exclusion zone around the line, within which only abstainers can be placed. Below are some examples of individual votes with the MPs now coloured according to how they voted (red = against, green = abstain, blue = for, grey = didn’t vote). The solid line is the vote line, the dashed ones showing the exclusion zone. In these examples (as with most), the visualisation has done a decent job of separating the fors and againsts (reds and blues), with predominantly only the abstains and non-voters appearing in the exclusion zone.





Rant: Politicians and YouGov polls

Louise Mensch is the Conservative MP for Corby.  Today she tweeted this:

RT @RicHolden: #YouGov polling numbers for female voters: #Conservatives 43%; #Labour 41%; LD 9%; Oth 7%. #feminism #VoteConservative

There is not a lot of difference between 41% and 43%.  The polling companies can’t feasibly ask every woman in the UK.  If you only ask a subset then small differences might just be because you happened to ask an unrepresentative sample.

Here is the actual poll data:

which says that they asked 1727 people.  That’s a reasonable number of people.  If you are statistically minded, you could work out if 1727 people are enough to make any conclusions about a 2% difference.

If you’re not statistically minded (or are, but are lazy), you can instead look at the same survey done just 3 days earlier:

and discover that three days ago, it was Lab 42%, Con 37%!

I can think of two reasons for this:

1: Women have suddenly (IN THE LAST 3 DAYS) started a mass Labour exodus (Ed Milliband has been pretty useless).

2: Polling a small number of people isn’t very reliable and making a big deal about it when it favours your particular cause makes you look a bit silly.

I’m inclined to go for the latter, although it would be nice to think that female support for UKIP is a fifth of what it was 3 days ago.  And that the 1% of males who would have voted for the BNP on the 6th Jan have since been culled/educated.

Two interesting features do stand out from the data:

1. Women are consistently more likely to give the answer ‘don’t know’ than men.  I don’t believe that women really ‘don’t know’ more (or that men ‘know’ more) so did a quick search to see if this is something that has been studied.  Could only find this (didn’t look for long), which looks at it for the quite specific question “do you snore” (!!): Correlates of the “don’t know” response to questions about snoring.

2. Older people become much less likely to say ‘don’t know’ as they get older.  Actually, I know some old people and this isn’t surprising.

How complex is the parliament?

I’ve finally got around to doing some analysis on the parliamentary vote data using a model that can handle both binary data (the votes) and missing data (MPs not voting).

Intuitively, it’s quite easy to imagine how it works:

Assume we have a load of MPs and one vote, that they all attended. They can only vote yes or no (1 or 0). It is clearly possible to get all of the MPs to stand in a line such that all of them to the left of some point voted yes (1) and all to the right voted no (0). We could then characterise each MP by their position in the line and from that could work out how they voted. We’d end up with something that looks like this (each digit is an MP):


Seem pointless? Yep, it would be. But what if there were two votes? If the MPs voted the same in both votes, it would be easy – we’d still only need one line, and one reference point. It would look like this (each MP is now a column, each row a vote):

V1: 000000011111111
V2: 000000011111111

If they voted completely oppositely, it would still be possible:

V1: 000000011111111
V2: 111111100000000

We can still use a single line if they vote a bit differently in two votes but two reference points will be required:

V1: 00000000111111
V2: 00001111111111

For vote 2, the reference point is slightly to the left of that for vote 1.

It’s just as easy to dream up voting patterns for which we can’t do this. For example, we can’t reorder the following MPs such that for each vote there is a point to the left of which they all vote 0 and to the right 1:

V1: 00000001111111
V2: 00011110000011

What we can do, is position the MPs in 2-dimensions – imagine placing them in a room rather than on a line. For each vote, we’ll split the room in two with a straight line such that all MPs on one side of the line vote one way, and all on the other side vote the other way.

For two votes, we could always position the MPs in two dimensions in this way (much like we could with one vote and a line). We might be able to do 3 votes in 2-dimensions, but we we can’t be sure – it depends on how they voted.

Given data for a full parliament (about 1600 votes), it’s interesting to see how well we can do this with a particular number of dimensions. For example, if we restrict ourselves to two dimensions, is it possible to lay the MPs out and draw a line for each vote such that each MP is on the correct side of each line? If it is not possible, what’s the highest number of MP-vote pairs that we can get right?

This might tell us something about the voting patterns of the parliament. For example, in the UK we have a 3 party system (or at least we have until recently). Lets assume that for each vote in say the 1997 parliament, Labour MPs voted one way, Conservative MPs the other and Lid Dems sided with the other two. If this were the case, we’d only need one dimension:

V1: 0000 1111 1111
V2: 0000 0000 1111
V3: 1111 1111 0000
V4: 1111 0000 0000

Each column is an MP, each block (set of four columns) a party (Labour Lib Dem Conservative, in that order). If we find for real data that we need only one dimension, it suggests this (or something similar) is happening.

Alternatively, if we assume that sometimes Labour and Conservatives vote together and the Lib Dems differently, we would need a second dimension.

The following plot shows the percentage of MP-vote pairs we can get right for different parliaments as we increase the number of dimensions:

(The line for 2010 should be treated with some caution – only 100 votes or so so far.)

To put the y-axis into perspective, in a parliament of 1600 votes and 600 MPs, an increase of 1% corresponds to getting approx 9000 additional votes correct – about 10 MPs worth if the MPs vote about 50% of the time.

The results suggest two things to me. Firstly, the voting patterns in each parliament are pretty simple – we can get a lot of the votes right with two dimensions. This is not surprising – most MPs will vote along party lines and we have three (main) parties. Secondly, it looks like the three successive Labour parliaments have been getting slightly more complex over time – 1997 seems to have the simplest structure, but not by much. Perhaps over time MPs became a bit more rebellious?


Finally had a moment to do the other parliaments for which I have data.  Here they are.  In each case the first plot is classic PCA and the second probabilistic:


The second axis is flipped in the PPCA plot – this happens in PCA – but it’s kind of interesting how there seem to be a selection of parties (PC, SDLP, SNP) on a line between Labour and the Lib Dems possibly representing a kind of political spectrum?  There also seems to be a huge wasteland splitting the Tories from everyone else.


Labour appear a lot less coherent than in 1997 (just comparing the probabilistic plots, I don’t really trust the spread in the classic ones because of the missing value problem).  Also, the line of small parties that seemed to link Lab and Lib Dem in 1997 has been pulled towards the Tories.  (Again, ignore the 2nd axis flip).

2010 (only 49 divisions so far)

So, not much data yet – only 49 divisions (votes) – the effect of the coalition is clear and it looks like interesting groups may be developing between the Tories and Lid Dems (we would need more data to be sure of this).  The coalition seems to have scared off the DUP (especially), PC and SNP who have all dramatically migrated towards Labour.

What next?

One criticism of this analysis is that PCA and PPCA both assume that the data are real-valued which is not the case for this data (each MP-vote pair is either -1, 1 or missing).  I’ve been pointed in the direction of PCA-like techniques for binary data and will post some results ar some point if I can get it to go a bit faster (there’s quite a lot of data!).

Politicians, visualisation, and missing values

British MPs do a lot of voting.  In an average parliament (if there is such a thing), there can be upwards of 1500 separate votes, also known as divisions.  The Public Whip is a brilliant site, dedicated to providing this data in an easy to use format.

The Public Whip also have a couple of nice little visualisations of the data, in particular the one here. This uses a mathematical techniques known as multi-dimensional scaling (MDS) to convert the specific voting pattern for each MP into two numbers that can then be plotted onto a map.  In other words, it lets us visualise the 1500 or so votes in a nice 2-dimensional plot.

To do this, one has to convert the votes into numbers.  Lets say, for arguments sake, that we use the number ‘1’ if the MP voted for a particular motion, ‘-1’ if they voted against it and ‘0’ for everything else (abstention, laziness, etc).  MDS isn’t the only technique that can be used for visualisation like this.  Chris Lightfoot used Principal Components Analysis (PCA) – a technique similar in mission to MDS but resulting in a different 2-dimensional plot (I’ve added an accessible-ish guide to PCA below – apologies to the purists).

I find this kind of exploratory analysis interesting and so got a project student to build some software that would allow the user to perform this analysis under different conditions (different parliaments, different sets of MPs, etc).  The result was a really nice bit of software that will hopefully be made freely available somewhere at some point.  My interest didn’t dwindle though and  I had a bit of free time the other day so exported some data from the application for the 2005 parliament to play with.  I used Matlab do a standard PCA on the 2005 parliament.  Here is a plot of the MPs in their new world, coloured according to party (some of the colours aren’t very clear…sorry!):

The two dimensions chosen by PCA nicely split the three main parties into separate clouds. This isn’t surprising: MPs will often vote with their party and so we’d expect MPs within a party to vote similarly and MPs from different parties to vote differently.

However, there seems to a strange centralising force!  Something is pulling politicians from all three parties into the centre of the plot, particularly obvious for the Conservatives (blue) whose MPs seem to form a line pointing towards the point (0,0).

It would be nice to think that this was the result of PCA capturing some interesting diversity within the main parties. However, as Chris Lightfoot discusses in his original analysis, this isn’t always the case: it’s impossible to disambiguate such real variation from artificial variation due to our representation of the data.

Recall that we gave each MP one of three values for each vote: -1,1 or 0.  It is this last category that is the problem.  An MP voting neither for nor against a particular motion could be doing so for a number of reasons, the most likely of which is that he or she wasn’t present on the day of the vote (play with this data for a while, and you’ll be struck by how rarely some MPs do vote).  Encoding this as ‘0’ has the effect of pulling MPs who don’t vote very often towards the centre of the plot.

Ideally, rather then being forced to pick a number, it would be better if we could treat it as missing.  However, PCA forces us to supply a number for each vote for all MPs.  Using ‘0’ (or indeed anything) is a problem.  Think of it this way: by using ‘0’, we are saying that the MP is sitting on the fence – halfway between for (1) and against (-1).  In some cases this may be reasonable, but for the most part, it’s likely to be a gross over-simplification.

In statistical terms, these values are ‘missing’ and the problem of missing values is widely studied in statistics because they occur all over the place.  Sometimes they occur randomly – imagine that the complete data-set exists somewhere and each value is kept or removed based on the toss of a coin. Other times there will be something systematic that causes values to be missing.  Either way, there is a lot of literature devoted to this.

Sadly, there is no wonderful solution to the missing value problem using standard PCA.  People use ad-hoc techniques like inserting average values or a constant value (that’s what I did with the ‘0’s).  Chris Lightfoot tried a couple of things that were, by his own admission, hacks.   They produced some interesting patterns but the problem with hacking around like this is that it’s impossible to be sure whether or not what we’re seeing means anything.

Fortunately, help is at hand.  Probabilistic models are popular within the data analysis world and one reason is their ability to handle missing values.  In fact, they not only overcome them, but can also simultaneously make an educated guess at what the values should have been – how the MP would have voted, had they been present.  One such model is Probabilistic PCA published by researchers from Microsoft Research in Cambridge (download it here – might be paywalled).  There is also what looks like an older technical from Sam Roweis (available here).  The details are all in those papers – mathematical literacy required!  Suffice to say that we can leave lots of MP-vote combinations blank and let the algorithm do its thing.

A bit of googling uncovered some code courtesy of Jakob Verbeek that does PPCA (sensibly, a la Roweis) in Matlab.  Here are the results:

The difference is striking – each party is now represented by a much tighter cluster. A tentative conclusion might be that most of the diversity observed in the previous plot was due to attendance, rather than voting patterns. This is a bit of a shame: I’d like my MPs to be a bit more independent, but that’s just my opinion.

There is some diversity in the plot, and we can probably be confident that this is real.  Look at the little tail of Labour MPs (red) going towards the Lib Dems. The fact that they’re joined by SDLP members is reassuring. I need to spend some time identifying some of these people to be certain that it makes sense but my guess is that it will. Other points of interest: in the first analysis, the DUP (black dots, hard to distinguish from blue ones) were intermingled with the Conservatives. It looks like this is just due to attendance: in the second plot, they form a nice little coherent group all by themselves – a kind of mini, more liberal Conservative party.

The only group that are now spread around are the independents (green dots) and there is no reason why they would vote together because, um, they’re independent.

At first glance, it looks like the PPCA is doing a great job – the grouping is incredibly clean.  When I get some more time, I’d like to look into this further – PCA (and PPCA) find more than 2-dimensions.  Looking at the third, fourth, fifth etc might provide some interesting patterns.  It would be nice to also look at other parliaments (data from 1997, 2001 and 2010) is available.  It would also be nice to look at the data for just one party.  Inter-party differences will always be greater than intra-party differences (that’s why they are different parties!) and so swamps the analysis.

If nothing else, it’s a nice dataset for the data analysis community to play with to test their algorithms for projection and missing value imputation.

Appendix: Hand-wavy introduction to PCA

Imagine that there are three votes, rather than 1000.  We want to combine these three numbers (remember that the votes are now numbers: 0,-1 or 1) somehow to give us 2 numbers for each MP that we can use for plotting – an x-coordinate and a y-coordinate if you like.

One way in which we could combine them is to add them up.  If a particular MP voted -1,-1,0, their new number would be -2.  More generally, we can assign a weight to each vote, multiply the vote by the weight and add up the results.  Say our weights were 1,-2,5 our number would become (1 x -1) + (-2 x -1) + (5 x 0) = 1.

We could pick as many sets of weights as we like to give us as many new numbers as we like!  This is exactly what PCA does – it picks 2 sets of weights such that for each MP, we can create 2 numbers from her voting pattern (actually PCA can get more than 2, but let’s not worry about that).

I picked those weights randomly.  PCA picks them systematically such that when we plot the MPs in their new 2-dimensional world, they are as different as possible. This is good, because it tends to reveal any group structure in the data.  If MPs vote in groups (parties), these groups will be apparent when we do the PCA.