Empirical research on citation patterns and scholarly reputation in academic research („scientometrics“) frequently relies on a commercial database called Scopus. Scientometric studies routinely analyse the contents of this database as representing the universe of research articles in various academic fields, including legal research. We reviewed the publically available data on German law journals and found that hardly ten percent of them are classified correctly. Even our most charitable count puts the misclassification rate at around 60 percent. In scientometrically studies regarding legal research, the use of Scopus is therefore indefensible.
Evidence of an indefensible research practice?
A first clue that one cannot trust quantitative findings on legal research that are based on Scopus data comes from the Scimago Journal & Country Rank (SJR) software built upon Scopus. It provides a journal ranking (in 2021) of 818 law journals worldwide, including 46 from Germany. According to the 2021 Scimago ranking, the highest-ranked German law journal in existence is…
Journal of Police and Criminal Psychology
Even if you are a German legal academic, you have probably never even heard of this journal. Which is not surprising, given both its tangential relationship with legal research and the fact that among its 23 editorial contributors, exactly zero are based in Germany. Further down in the ranking, we find titles such as Asian Journal of Law and Economics or Jindal Global Law Review (German, really?), as well as Danube and Credit and Capital Markets (Law, really?).
Why would these journals still be classified as German law journals? Regarding the „law“ category, there are of course legal implications of capital markets research. Or of social science research in the Danube region, for that matter. These implications are hard to capture objectively, so we will not quibble over the subject matter categorisation. What can be objectively determined, though, is a journal’s geographical location. In this case, Germany.
Validating Scopus data on „German“ law journals
You could think of several interpretations of what it means for a journal to be „German“. You could look at its language (which is still the most prominent marker of regional identity in the discipline of law), or its primary physical contact address, or the affiliation of its editors – or all of these combined, which is the approach that we settled with. Scopus, on the other hand, seems to rely exclusively on the address of the journal’s publisher. Yet, this is hardly what scholars mean when they refer to a „German“ journal, and it cannot even explain why Scopus would classify European Journal of Risk Regulation (published by Cambridge UP) or Political and Legal Anthropology Review (published by Wiley for the American Anthropological Association) or Danube (published by De Gruyter Poland) or Connections (published by Procon Ltd. in Bulgaria) as German in the first place.
To get a better sense of how the Scimago-ranked „law“ journals relate to Germany (and to quantify the actual misclassification rate), we manually reviewed all journals contained in the Scimago ranking (and, by extension, Scopus) for the search criteria „Subject category: Law“, „Region/Country: Germany“, and „Year: 2021“:
For each of these 46 journals, we coded whether the journal (not the publisher) provides a contact address in Germany, whether it accepts German-language articles and how many of its editorial contributors state a German affiliation. These are the results:
Four journals (8.7 %) bear no discernible relationship with Germany at all. Meaning their contact adress is outside Germany, they do not publish in German and do not have a single editorial contributor with a German affiliation. While two of these four journals are at least published by a publishing company in Germany, the other two are not. We can only speculate why they would be listed as German law journals at all.
Another ten journals (21.7 %) bear the least imaginable relationship with Germany. Meaning that either they maintain a letter box in Germany, or they have a single German editor (out of up to 60, in the case of European Journal of Risk Regulation) – but not both. Arguably though, a German editorship of 1.7 % does not a German journal make.
Another 17 journals (37.0 %) do bear some personal relationship with Germany because up to 40 % of their respective editorship is based in Germany: These journals count between one and seven German-affiliated scholars among their editorial contributors, and all but one (Political and Legal Anthropology Review) are contactable at a German address. We would still argue that a law journal is hardly „German“ merely by virtue of its letter box and much less than half of its editorship (even less than 3 % in three cases!). Yet, four of these journals actually accept articles in German, which convinced us to qualify them as German journals after all. Those are Accounting, Economics and Law: A Convivium, and Zeitschrift der Savigny-Stiftung fur Rechtsgeschichte (two journals), as well as Journal of Intellectual Property, Information Technology and E-Commerce Law (JIPITEC).
Like these four, the remaining 15 journals (32.6 %) bear an overall convincing relationship to Germany: In each of these cases, at least 45 % of their editors state a German affiliation, and 11 of these 15 journals even feature a German title. Note that the other four, however, don’t even publish articles in German: One could wonder, therefore whether Credit and Capital Markets and International Journal of Conflict and Violence should be considered „German“ „law“ journals (while the German Yearbook of International Law and the Max Planck Society’s International Review of Intellectual Property and Competition Law surely should).
Even if we silence any doubts on account of these journals‘ strong personal ties with Germany, and if we then also count the four journals that do publish German articles from the previous category, this highly charitable counting method allows us to qualify no more than 19 out of 46 journals (41.3 %) as even superficially German.
Overall, merely a one-tenth reliability rate!
For anyone familiar with German legal academia, here are these 19 journals, sorted by Scimago’s proprietary ranking metric:
3. IIC International Review of Intellectual Property and Competition Law
7. Accounting, Economics and Law: A Convivium
8. Journal of Intellectual Property, Information Technology and E-Commerce Law
9. International Journal of Conflict and Violence
13. Forensische Psychiatrie, Psychologie, Kriminologie
22. Kriminalistik
23. Credit and Capital Markets
25. Deutsche Lebensmittel-Rundschau
28. Monatsschrift fur Kriminologie und Strafrechtsreform
30. Recht und Psychiatrie
31. Rechtsgeschichte
33. Archiv für Rechts- und Sozialphilosophie
40. Kriminologisches Journal
41. German Yearbook of International Law
42. Verwaltung
43. Zeitschrift fur Rechtssoziologie
44. Zeitschrift der Savigny-Stiftung fur Rechtsgeschichte, Germanistische Abteilung
45. Zeitschrift der Savigny-Stiftung fur Rechtsgeschichte, Kanonistische Abteilung
46. Zeitschrift der Savigny-Stiftung fur Rechtsgeschichte, Romanistische Abteilung
Now compare this with your intuition of the most cited, most prestigious or simply well-known journals in any area of German law that you may be familiar with. Any overlaps? We saw three. Maybe four. (Plus the three Savigny journals, which are leading venues of legal history.)
Long story short: We barely find a handful of arguably-German, arguably-legal and arguably-well-known journals in Scimago’s ranking of German law journals. That is roughly a tenth of the 46 journals listed in that database. In our view, therefore, no scientometric study referring to legal research should use these data, and legal researchers should call such studies out for what they are – not nearly representing legal academia, but based on an extremely eclectic convenience sample. Scopus‘ collection of 46 „German“ law journals seems to be due in no small part to efficient marketing by the (German) Walter de Gruyter GmbH publishing company, whose subsidiaries (anywhere in Europe) publish 19 of these journals. Does that mean they represent German legal academia? No way, José.
Note: Our manual codings of these 46 journals, as well as their raw data from Scimago, are available on Zenodo.
Distributed under the Creative Commons License CC BY 4.0
