Disclaimer. Don't rely on these old notes in lieu of reading the literature, but they can jog your memory. As a grad student long ago, my peers and I collaborated to write and exchange summaries of political science research. I posted them to a wiki-style website. "Wikisum" is now dead but archived here. I cannot vouch for these notes' accuracy, nor can I even say who wrote them. If you have more recent summaries to add to this collection, send them my way I guess. Sorry for the ads; they cover the costs of keeping this online.
Moser. 1999. Electoral systems and the number of parties in postcommunist states. World politics 51.
Helps to rectify Cox (1997 and 1999), who argues that electoral rules can affect the number of parties, with Mainwaring and Scully (1995), who argue that institutionalization of the party system matters before the number of parties can. Definitely see also Reilly, who, like Moser, argues that certain rules (in this case, the alternative vote) can help a stable party system to form where there currently isn't one.
Cox argues that electoral systems can be expected to lead to party consolidation when three conditions hold. Among these is that we have preferences over the existing parties, and that we have an idea of which parties have the most realistic chance of actually winning a seat. According to Moser, these two conditions could be stated differently: Cox's theory holds only when party systems are institutionalized (i.e. in consolidated democracy where there is a relatively stable party system). However, in the early years of post-Communist democracy, party institutionalization cannot be assumed.
Thus, PR (with a threshold) can do a better job of producing lasting, consolidated parties than single-member districts: PR forces parties to coalesce and endure (at the national level) while SMD does not (candidates can run without a party in SMD, thus not coalesce into parties). In a sense, then, Cox's theory should be this: In established democracies, SMD promotes party consolidation the most; in new democracies, PR promotes party consolidation the most.
Compares five post-Communist states. Of those states with a two-tiered legislature, he compares each tier as a separate case:
Evidence is in Tables 2 and 3 (compare them to see effect of PR vs SMD).
Research on similar subjects