Our key question driving this validation study was: are UIC-based data suitable for assessment and comparison of universities within the context of university–industry R&D linkages? To address this question our cross-validation study examined the information value of various UIC-based measures—either as stand-alone metrics or integrated into composite indexes. We adopted Thomson Reuters’ Ranking of Innovative Universities (RIU) as an external frame of reference to assess their analytical relevance and statistical robustness.

When used in a stand-alone mode, UIC-based metrics should ideally comply with the newly-minted concept of ‘responsible metrics’, featuring in the recent ‘Metric Tide’ report (Wilsdon et al. 2015), and it associated list of quality criteria in which ‘robustness’, ‘transparency’ and ‘diversity’ are highlighted as key attributes. This report’s recommendations state that “indicators and metrics are based on the best possible data in terms of accuracy and scope”, “data collection and analytical processes are as open and transparent as possible, so users can test and verify results”, and “indicators and metrics to reflect and support the diversity and plurality of university performance features”.

What do UIC-based metrics represent, and how do they relate to other metrics of university ‘innovation’? Our study has demonstrated that UIC metrics are able to capture ‘diversity’ among universities: UICs represent different types of R&D-related proximity relationships, which are not easily disentangled, as well as other ‘knowledge flow’ related phenomena. UIC counts also capture size effects, and possibly even scale effects, among universities—where a few large US universities are consistently ranked among the highest worldwide.

Should one apply size-dependent or size-independent UIC-based metrics? Including size-dependent measures has major implications for positions of universities in the Thomson Reuters’ ranking. Many less prolific universities, i.e. with smaller quantities of research publications and/or patents, are significantly disadvantaged. Our findings show that this is particularly detrimental for the high-performing Japanese universities as well as some Western European universities, which are all ranked in the top 25 according to the U-I R&D Linkage Index (see Table 9). Clearly, whether or not to incorporate a university’s size is one of the key determinants of its ranking position. The same applies to single UIC metrics: applying the absolute number of UICs produces a completely different picture compared to the share of UICs in university’s publication output. While offering the advantage of computational transparency, their reliance on a single source of information reduces their analytical scope. UICs alone are clearly an insufficient measure of university–industry R&D linkages. These co-authored publications are just one of many measurable linkages between universities and the business sector.

Does that make data-heavy, multiple-metrics indexes superior analytical tools? What are the (dis)advantages when several metrics are combined or integrated into a composite measure? Composites clearly offer ways of incorporating a wide range of sources and metrics, thereby reducing negative effects of source-specific and country-specific differences between universities, and thus providing a more balanced and robust measure suited for comparative purposes. Composite measures, partially based on UIC-metrics, are therefore a superior proxy, provided both absolute and relative size needs are factored in. Although this might enable us to identify top ranked ‘powerhouse’ universities, no index alone can provide a fully adequate assessment of an institution’s performance. Any index-based university ranking will still depend significantly on the selected metrics, the computational method that was applied to assembled in a composite measure.

This tradeoff between single-metrics and multiple-metrics needs careful navigation in search of an optimized approach. Inevitably, differences are to be expected and significant discrepancies between ranking positions may easily arise. Usually as a result of an interplay between the choice of metrics, the computational methods applied, and because of differences between information sources.

Our sensitivity tests indicate that the various UIC-based metrics and indexes are not consistent proxy measures: in most cases a university’s spread of ranking positions across the between the various rankings is simply too large. Although both index-based rankings in this study are, in a sense, useful representatives of U-I R&D linkages, there is no formalized rationale or intuitive heuristic to opt for either as ‘the best’ representation. Nor is there a ‘one-size-fits-all’ measurement model. Our findings also highlight both conceptual and computational weaknesses in the RIU. Such top-down producer-driven rankings, with a single composite score based on questionable weighting systems, are suboptimal data reduction tools. Clearly, the trade-off between data reduction and generating meaningful outcomes is problematic and in effect disqualifies these league table rankings as representing ‘best practice’.

Can one develop a ‘good practice’ to develop more robust and meaningful university ranking systems? Based on this study’s outcomes, we recommend a bottom-up ‘scoreboard’ approach without those problematic data reduction constraints. This user-driven approach has already been adopted in the Leiden Ranking and U-Multirank, both ‘open access’ information tool where: (1) no composite measures and weighting systems of metrics are implemented, (2) computational procedures and information sources are more transparent, (3) size-dependent metrics are kept separate from size-independent metrics, (4) performance metrics are selected according to proximity relationships between universities and industry. Users should be able to (de)select metrics of their choice according to their own selection criteria and analyze the scoreboard data anyway they like to satisfy their information requirements.

Irrespective of the measurement model, choice of metrics, and the type of ranking system, the availability of such information has opened up a new comparative framework with implications for various major stakeholders.10 First, the ranking producers themselves: here we expect to see follow-up publications of ranking data. On June 14th, 2016 Thomson Reuters has published the list of ‘Europe’s top 100 innovative universities’.11 More university rankings dealing with aspects related to ‘innovation’ (or related topics such as ‘entrepreneurship’ or ‘industry orientation’) will surely appear in the years to come. Secondly, the universities where one may expect to find the same institutional response as in the case of other rankings: high scores are likely to be publicly acknowledged on university websites or in press releases; low scores will be either ignored or perhaps used in management decision-making on the strategic development of a university. The popular press is a third stakeholder: any newsworthy information introduced by university rankings is of interest. More rankings creates more opportunities for news articles and blog posts.

These new ranking systems on ‘innovation’ may also create applications in government policy making domains. Although we are unaware of such policy impacts to this date, we may speculatively assume that such rankings will indirectly affect policy making at the national or regional level. Those impacts could emerge in terms of offering a new source of evidence for policy debate or raising the awareness of such performance measures for policy recommendations. It seems much less likely that rankings, given their still questionable robustness, will be used to support resource-allocation decisions—either to address perceived shortcomings in university–industry R&D linkage relationships or to boost the performance of individual ‘innovation-oriented’ universities.