S1E10.1 - Publish or Perish (Academic Publishing)

#Academia #Publications #Satire #PublishOrPerish #BoardGames #Science

Introduction

Merry early Christmas, as we go through a short bonus episode on Publish or Perish, a satirical card game by Dr. Max Bai. We talk about the nature of academic publishing, including problematic aspects like predatory journals, and how some of the quirkier aspects of the process get reflected in the game. So enjoy this lighter offering, and we'll see you with Season 2 in 2025!

Find our socials at https://www.gamingwithscience.net 

Timestamps

• 00:00 - Introduction
• 01:53 - The Importance of Publications
• 05:06 - Gameplay and Mechanics
• 12:15 - Grades (& more Importance of Publication)
• 15:52 - Generative AI & Predatory Journals
• 21:26 - Wrap-up

Links

Publish or Perish (Kickstarter page) 

Full Transcript

Brian  0:06  
Hello and welcome to the gaming with science podcast where we talk about the science behind some of your favorite games. In this bonus episode, we're going to discuss publish or perish by Dr Max Bai

Jason  0:20  
All right, everyone, welcome back. We're well, Brian says that this one is actually science with gaming instead of gaming with science as we're doing a game about the scientific process itself instead of about a science topic. This is Jason. This is Brian. And welcome technically, we're in our between season break right now, but we like giving y'all bonus episodes, and honestly, this was something we couldn't pass up. So we're just going to dive right into it. Publish or Perish, is a game that is just out by Dr Max Bai. So he is an independent social psychologist. So he got his PhD in social psychology, did a postdoc at Stanford, and now runs an independent research lab, which I don't know exactly how that works. He says on his Kickstarter page he started a few companies, so I assume they provide him enough income he can just do his own research what he wants. And maybe social research is less expensive than biology research. I don't know. I do not have enough money to be an independent researcher and run my own lab.

Brian  1:07  
Well I mean, research can vary widely in how expensive or inexpensive it is, depending on how you're doing it.

Jason  1:13  
but whatever the case is, he's running an independent research lab, which means he doesn't have any of the administrative overhead and all the deans and stuff that we spend all our time complaining about, and that's probably a preview for how this episode is going to go, because this episode is about the scientific process itself, not really about any specific scientific discipline. And so you're going to see maybe a peek behind the curtains, if you don't know it already, if you're already in the sciences, then hopefully this is not too traumatizing, as we bring up maybe some of the less fun parts of being a researcher and a scientist. So anyway, what is this game? I don't know if he made this as a graduate student or as a postdoc. It is a light party game meant to kind of poke fun at the scientific publishing enterprise. So we've mentioned this a few times on the episode. We scientists don't have very much. We're generally not in it for the money. Most of us don't get very famous like the one thing we have is our reputation with other scientists, and we establish that by publishing academic research papers. And people look at those papers, and that's how things like promotion and tenure, which is basically job security and being hired by another university or going off into industry is important, like if you're going on the job market either as a freshly minted Master's or PhD student, or as a professor who's been in it for 20 years, people are going to look at your publication record to see, are you actually a good scientist? Are you actually putting out a lot of work, and hopefully good quality work. But as what happens with any field, anytime you reward people for something, then there's ways to sort of abuse the system, and things go wrong, and the publish or perish of the title is talking about how, as a researcher, especially a university researcher, you have to publish or you'll perish, like if you don't get the publications out, you're going to lose your job. 

Yeah it's kind of one of the unfortunate realities that, like science can be thought of as this very pure thing, the pursuit of knowledge, the generation, the advancement of human knowledge and humankind. It's also a job that's kind of difficult. And the unit of scientific information that your typical scientist cares about is publishing a paper. We did a study. This is what we found. You take that paper, you write it all up, you submit it to a journal. It'll go through a process called peer review, which is where three anonymous colleagues typically will look at it and say, like, is this good science or not? And that process kind of works. It's, I think it's like democracy. It's the worst political system except for all the other ones.

Churchill's famous quote, 

Brian  3:26  
yeah. 

Jason  3:26  
So this system has evolved over a century or two of time as the scientific enterprise kind of started building as we recognize it now. And no, it's not perfect by any means, and people are looking at ways of changing it. So there's like, open peer review, where you publish and then people just comment on it, which has all the pros and cons of everything else on the internet that's just open for people to comment on, 

Brian  3:47  
just like Wikipedia, right? 

Jason  3:49  
Yeah, peer review, as currently, as it currently stands, also has the same pros and cons, because peer reviewers are usually anonymous. And one thing the Internet has taught us is that sometimes when you have anonymity, you get permission to be nasty, and some people do that. And that is actually the infamous reviewer two that the game talks about a few times is that it's usually it's reviewer number two is that nasty person who just hates your work and didn't read it, and is just saying how much horrible stuff is in there, and you have to change all this stuff and so on and so forth.

Brian  4:16  
Being on the other side of this, you know, being someone who does publish papers in almost everything that we do as scientists, you have to justify what you're saying. You either did the experiment or you can cite someone else who did the experiment. You can cite something that's passed peer review, that's out in common knowledge. When you are an anonymous reviewer, people will just say random stuff that's not justified. They don't provide their sources. They don't have to explain anything. And that's one of the only times that scientists get away with this crap. 

Jason  4:43  
And I will say this is not like this isn't every time, 

Brian  4:46  
no, no, no, 

Jason  4:47  
but it's not rare enough that it's a weird thing. Like most people have had an experience with THAT reviewer who just didn't get their thing, 

Brian  4:54  
and then you usually have an editor who's kind of acting like a referee. They will read the paper, they will. Also look at the reviewers comments and say, like, hey, is this person out of line? They're the, pretty much the only people that get to say that.

Jason  5:06  
But we don't have any of that in the game. In the game, all you have to do is you get enough cards to get to publish. So let's actually talk about this game before we dig into the nitty gritty of scientific publishing. The game has a few different types of cards. You've got your action cards, or what you draw into your hand and you used to play the game. There are publication cards, manuscript cards, which are what you collect to try to win the game. Each one's worth a certain number of citation points, because that's how we grade papers in science. Is how many times do other people cite the paper? And then there's some trivia cards, actually, that get used in a few cases, where they have random trivia from all sorts of different fields of science. And the experience there is, if you are in that field, then they're easy, and if you're not, they're super hard.

Brian  5:46  
We experienced this. It's like, oh, these biology questions are way too easy. I don't even know what these terms are in the economics question, 

Jason  5:53  
yeah, which is probably exactly what it's meant to be. They're meant to be high level stuff, so that if you're not in the field, it's basically just random guessing. And the idea is that you go through, you're drawing your hand, you're collecting cards. And if you get certain numbers of the right resources that you can then get a manuscript. And there are five resources which are basically there just to have five different things you have to collect. There's writing, theory, references, data and ideas, and so different papers require different numbers of each of those. Usually they just require, like, two or three of some of them. Honestly, the manuscript cards, I think are the best part of this game, because they're all fully fleshed out with a journal and an article and a title and an abstract, which is like a brief summary of it, and they're just ridiculous. So I'm just going to flip through here. We've got the economics of Santa Claus, an analysis of infinite resource management procrastination patterns among academics, a case study of myself unpacking the aerodynamics of flying pigs. And then there's a whole paragraph describing what this article is about, and he does say he used a little bit of generative AI to help with that, because I know there's like, 50 or 60 of these cards. I don't know how someone would come up with all that for all of them. So they're actually quite fun to read, because they're just totally ridiculous.

They've all got punny author names as well.

So I'm just picking the top of the stack here. Got a myth busting microwave minutes is defrost, just a placebo, and the author is luke warm. SCD from the University of convenient conclusions.

Brian  7:12  
You can also hit your opponent with maladies, budget cuts, which will affect things, or a citation error, which will remove some of the citation values. And those all have funny little quips on them too.

Jason  7:22  
Yes, they actually the the flavor text on the card. So not the mechanics. The mechanics are pretty standard across a lot of the card types, but the flavor text of what exactly went wrong is actually quite entertaining. So there are mishap cards which you play on other people to reduce their citation counts. Just grab one here. As it turns out, a critical citation for your paper was hallucinated by chat GTP, that's your thing, and then you have to spend resource cards to fix that error so you can get your citations back. 

Brian  7:47  
Have you ever experienced this? Because I know that this was the thing in the news where chat GPT will fully manufacture scientific articles, the authors, the title, the date, everything. It'll even give you fake Digital Object Identifier URLs for papers that do not exist.

Jason  8:02  
Yes, I tried that early on. It's like, okay, let's see how this is. Write me a short scientific literature with citations for whatever topic I was looking at the time. And it did it. It wrote a very nice paragraph, and it had citations. And I looked them up, and they were all completely non existent. 

Brian  8:17  
They sounded real.

Jason  8:18  
 I've noticed they've started adjusting that now where they'll actually have, like, footnotes, maybe not on chat GPT, just different AIs I've used, they'll put footnotes as where this information came from, so you can actually look up your sources. And there's a few where they very specifically were making it for scientific research, where they would have, like, this sentence came from this paper that you can then look up. So they're trying to fix that problem 

Brian  8:39  
anyway. This is a different conversation. 

Jason  8:40  
Yeah, different one. So game, back to the game. So, and your goal is just to get the first one to five citations ends the game and you finish out the round. But they don't necessarily win, because you have to get your citation count. And then we didn't do this part because it was late and we were tired. 

Brian  8:55  
Well, it's also only two of us. This game says it's got a player count of three, because it's a party game. We, I mean, we just did it because we've got the game for a week and we wanted to play. Wanted to play it right.

Jason  9:03  
Yeah, so we had to review an early review copy that we have to send back in a few days. But whenever someone publishes, everyone else is supposed to clap and then congratulate them or give snarky comments and reviews phrased in the form of a question, like you were a reviewer trying to tear apart their publication, and at the end of the game, there's actually a vote for whoever was the snarkiest reviewer, and they get extra points. You also take all the publications you managed to collect and put together basically a fake dissertation defense, where you're defending your line of research with these random papers you've cobbled together. And that goes on for supposedly one minute, although usually longer, and then you get a vote, and whoever did that the best also gets additional points. And then there's the "almost there" award, which is for the person who tried to get those previous two and failed. And so they get, instead of three extra citations, they get 2.9 extra citations. So it's, it's, this is definitely a party game. It's a light fluff game. This is not deep gameplay. There's not deep strategies. To explore here. It's just you gather some people around the table, you sit down, you play, you have fun. Obviously, hit strongest if you're in academia, so like, if you've been through a graduate program or stuff, but you don't have to do that to play. Like, you can kind of get it, and the cards are funny enough that you can kind of get what's going on without that. But I think its original target audience was definitely academics. 

Brian  10:19  
Didn't Didn't you say that this kind of started making the rounds because it got some attention in Nature magazine. 

Jason  10:23  
I think it got some attention on social media, and then it got picked up by Nature, which is one of the big scientific journals, and so lots of people see that. And so, yeah, it got within at least the scientific community. It got a lot of traction. 

Brian  10:36  
I mean, I had two people send it to me. They're like, Hey, you should talk about this. Did you have that as well?

Jason  10:40  
I did not. Maybe I short circuited, because as soon as I found it, I sent it out to other people, so maybe 

Brian  10:46  
they didn't have the opportunity. 

Jason  10:47  
Yeah, 

Brian  10:48  
well, by the time they were sending it to me, you had already done that as well. So actually, I think I got it from three or four different people, you included. I think the best player count for this game. Party games are always better with more people. What's the max suggested player count?

Jason  10:59  
Let's see. So the game says it is for somewhere. I think it's three to six, yeah, three to six players. Says it lasts anywhere from half an hour to two hours. 

Brian  11:09  
Two hours? That seems crazy. 

Jason  11:11  
If it's really cutthroat, maybe, I don't know. 

Brian  11:13  
I mean, we played two rounds in an hour. There were only two of us, though, 

Jason  11:16  
yeah, and it was fun, like it was a nice, quick little fluff game. I'm gonna be playing it with my lab over lunch today, so we'll see how that goes, and I'm going to try to get them to do the improv and the sillier parts that you were not on board for

Brian  11:26  
improv is more fun when it's not just one person across the table from you. So 

Jason  11:30  
that is fair. 

Brian  11:31  
Yeah, I think five to six is going to be a lot more fun than three. 

Jason  11:34  
Yeah, Isuspect because you have more people to riff off of, there's more chance to be snarky because you're not like you're not always snarky. I can't come up with a snarky comment for every paper, but the more people there are, the more likely that is to happen. I've already pre ordered a copy of this that I'm gonna have in the lab just to blow off some steam or do over lunch or take to department retreats or something. And on the Kickstarter, not only do they have the core game, typical, they already have the first three expansions made. One just adds extra action cards. One sets extra trivia cards, and one adds extra manuscript cards, including predatory journals that will basically publish anything you give them. And in fact, one of them, the abstract is lorem ipsum, which is this Latin text that is basically a fill in for editors. They just use to fill space. I think that the title of that one was scientific evidence that predatory journals will publish anything you give them.

Speaker 1  12:18  
which believe it or not, that has been an actual thing, like, there have been a couple studies on that very topic for real in real life. Maybe I'll find a link for one of those.

Jason  12:25  
Yeah, so that's really the game. I mean, it's a light game. It's a fun game. Looks like when it goes live, probably by the time we get this edited, Kickstarter is going to be done. Sorry about that. There were some delays in getting us our review copy, but it is going to be available commercially after that. Looks like the retail price is going to be about $40 for the base game, and somewhere around 20 for each expansion you look out for that either online. I'm sure he'll have it available online somewhere, maybe managed to get into some big box stores or local game stores or something. 

Brian  12:51  
What do you wanna do for this one? So this is a is this getting a science grade or just a fun grade?

Jason  12:55  
I think mostly it's getting a fun grade because, I mean, if we were trying to grade on what is this like according to actual scientific publishing, actual scientific publishing is a grind and is generally not fun.

Brian  13:07  
Yeah, stretch this out, where each round you submit, and then you wait for about four months to hear something back.

Jason  13:13  
 And the thing is, like, we're making this sound bad, and part of it's because most of us do not go into science because we love writing papers. We go into science because we love doing the science, and we love doing experiments and solving problems and writing the paper has to happen, but it's kind of like doing your taxes or otherwise filing forms. It's like, it's not necessarily the fun part of what we do,

Brian  13:33  
but super critical. It's the whole thing, right? I mean, you bring in money to produce science, and this is how we do it. You gotta publish it. You gotta get it out there. Or what was the point of doing it?

Jason  13:42  
Yeah, unpublished sciences. I mean, technically, it is science, but doesn't it's not part,

Brian  13:46  
 it's not useful.

Jason  13:47  
 Science that doesn't make it into the public consciousness, that doesn't make it out there for other people to use, is, I wouldn't say useless, because that's companies do that all the time. It's proprietary, but we're in the public sector. Our job is to create knowledge for the public good, which is why people in industry publish less, not zero. So industry researchers can actually publish papers on what they are doing. They usually have to go through some hoops to lock down intellectual property first, because a company's goal is to get a competitive advantage, but if they can do that, then they can put it out, and it does actually boost their reputation some too, especially if there's like a tool they've developed that lots of people would be really interested in using they want people to know about it so that they can license to them and make money off of it that way. So he even has a role in the private sector, just not as much a role in the public sector.

Brian  14:29  
OK, so we're probably gonna skip giving it a science accuracy grade, maybe just to protect my own mental health. But in terms of fun, I probably think for the right audience, this would be an A, but if you're thinking about a general public thing where their lives are not touched by the scientific publishing endeavor, I don't really know if this is gonna hit. So am I allowed to give a split grade?

Jason  14:51  
This is a bonus episode. You give whatever you want, 

Brian  14:53  
all right, if you have had science publication, or know someone who has in your life, I think this will probably be an A. And if you don't, I think it's probably a B-. I don't think the jokes are going to hit all that well.

Jason  15:04  
yeah, hopefully they'd enjoy the manuscript pages, though, the abstracts and the titles are actually quite fun. The names are funny. Being one who's in academia, I'd probably give it like A-, B+. But I like crunchier games. So there's very few party games that I just enjoy sitting around and playing with people, because I prefer more ones where there's tactics and there's some rules I can try to figure out how to master, and my advantage comes from being able to master those rules better than other people, rather than just doing goofy stuff with my friends, which there's a place for that. My family loves Cards Against Humanity, which is definitely just a light, fluffy party game. But most of the time I prefer something with more meat to it, but that's my personal preference.

Brian  15:36  
all right. Well, I mean, we're gonna skip the science again, just because it hurts my heart too much to have to get into the details here. 

Jason  15:42  
It almost feels like we did this episode backwards. We talked about the actual stuff before we talked about the game. 

Brian  15:47  
Well, I mean, but it is a backwards episode. It's the science with gaming.

Jason  15:52  
Here's something else I want to talk about, though, and that's actually the use of generative AI to make these things and the thing is, there's a big conversation now, my wife and I have argued back and forth about what's acceptable use of AI and not we have very different opinions on that. We're not going to get into that more, just the fact that the genie is out of the bottle its going to happen. So you mentioned that generative AI can create an entire fake scientific paper, and the game actually gets a little bit into like the predatory Journal, the ones that basically make money off of people paying to publish in them, because they need publication counts to let anything through. And my concern is like, where are we gonna go with generative AI, for this in the future? Because it used to be that writing a paper was really hard, even if you're a bad actor and you lied about your results, it took a lot of work to put a paper together and get it through the publication process. But now generative AI, makes that easy, and you combine that with predatory journals, and I just worry that the scientific literature is gonna get pollutedwith a bunch of crap papers, that's a big problem, especially when people try to reproduce what you do, which is a key part of science, one publication means nothing. 

Brian  16:47  
It's a body of work. 

Jason  16:49  
Yes, like one publication is like, okay, that's an indication, but until other people also repeat it and get the same result, then it's just, it's a data point. It's not actually considered real unless it can be replicated.

Brian  17:00  
Although I would say that there's a problem with that idea, though, because you're right. It is supposed to be part of the process, but replication studies often don't happen because there's this unfortunate human rationale of like, well, that's already been done. I'm not going to waste my time doing it. So the problems come when someone tries to build on previous work and then it doesn't work. Like, oh, wait, something's gone wrong here. That also faces the publication filter. People are less likely to publish the results that don't, quote, unquote, work. So it should be a self-correcting process, but the sort of messy human sociology, this is my jobness of it very much, can get in the way of that process. 

Jason  17:32  
That's true, and that's kind of where I'm concerned here, just like with everything else, with generative AI, it's possible that the noise will start swamping out the signal, because there are definitely people who are motivated just get a publication count out there, because that's what will be used to boost up their citation count. It might get them a bonus at their job. There are some countries that very specifically tie publications to your salary and to your promotion. And anytime you get that sort of perverse incentive, then you start encouraging people to cheat. Hopefully most people won't, but there will always be some number of people who will. Scientists are still human, and that means there's a whole bunch of different types of scientists, and some of them are going to game the system because it helps them get ahead, unfortunately. 

Brian  18:13  
So predatory journals. Why do those exist? 

Jason  18:15  
Ah, so we have to go into a little bit of background of publication for this. So it used to be that you would send your manuscript to a big publishing area, they'd go review it, assuming it passed all that and got published. They then made their money off of selling subscriptions to universities and companies and such. Well, starting probably 20 years ago, maybe more is this open access movement, because lots of people can't afford to pay for that. And the idea is, if we're making this the common domain of knowledge for humanity, then humanity as a whole should be able to access it. So people started making results open access, where you could just you could access them no matter who you are. You didn't have to pay for them or anything like that, and that's good. But the fact is, it still costs money to to have a copy editor to make things look nice, and so the money has to come from somewhere. And so now, when you have an open access publication, the person publishing it is paying the cost of that. They're paying essentially the cost of making it look nice, of running the peer review process and all that sort of thing. And once that happened, there are some malicious people figured, hey, we can turn this into a business of having people pay us to publish their work. And good open access publications still keep a high quality standard, and they still make sure it gets filtered. And bad ones who are out just to make a buck don't. They're just using it as a way of getting money from researchers, many of them who don't know better, because they don't realize the publication is predatory, because, like with all scams and phishing attempts and such, they're very good at hiding and looking like something legit, and so we have this whole issue going on. Maybe this is a different topic, but the same thing happens with conferences. The other way we get our work out is not just papers, but we talk about them at conferences. And there are predatory conferences out there. 

Brian  19:46  
They're 100% are predatory conferences. I get invitations for "conferences" all over the world all the time,

Jason  19:52  
and so this is, maybe, I hesitate to call it, the seedy underbelly of science. It's more just the unfortunate reality of, as Brian said, science is. A job. Science is a human endeavor that costs time and energy and money, and there are rewards attached to it. And anytime you have that, there can be some people gaming the system. And the reason for talking about this is not to like doom and gloom and Oh, science is horrible. It's like, no, no. It's like, most people involved are good actors most of the time. It's okay, but it's imperfect. And some people have pointed out that there's an idea of scientism nowadays. Of like, oh, science is great. We must follow everything. It's like, recognize it's still a human endeavor. It's imperfect. 

Brian  20:27  
Look at the history of science, it will show you how much humans can put their fingerprint on data.

Jason  20:33  
Yeah. And the thing is, I think on the whole, it still is one of the better systems we have for knowledge discovery, and betterment like you compare how we are now to 500 years ago? A lot of that is due to scientific advancements. Just before this, we talked about Pandemic and our lifespan, infectious disease control. All of that is due to scientific stuff, where people have taken things, some of it from indigenous knowledge that got tested in more extensive ways. Some of it from pure Western scientific research from all over the place. But it got tested, it got validated. It went through the system. And so yes, the system makes mistakes. The system is imperfect, but it does more good than it does harm by and large. And so we don't want to under cut it, but we do want to make it more realistic. The game pokes fun at science, but it does in a way that is based in truth. I mean, all the best satire is based in truth. And so while we may be talking a bit of a downer in terms of like, Oh, these are some of the flaws in the system, this is also a good safety valve. Yeah, we know things are imperfect, so this is a good way of like being able to laugh at the imperfections and then maybe move on and try to make them better. 

Yeah, best not to ignore that there's a problem that was a lot of deep philosophy for a game that is primarily based on puns,

puns in a fluff party game. All right? Well, this is just a quick bonus episode, so I think we're gonna call it then. So look up publish or perish by Dr max by hope you can have some fun if you decide to go for it, and until then, have a good break. Happy gaming, and we'll see you next time.

Brian  21:48  
Have fun playing dice with the universe. See ya. 

Jason  21:52  
This has been the gaming with Science Podcast copyright 2024 listeners are free to reuse this recording for any non commercial purpose, as long as credit is given to gaming with science. This podcast is produced with the support from the University of Georgia. All opinions are those of the hosts, and do not imply endorsement by the sponsors. If you wish to purchase any of the games we talked about, we encourage you to do so through your friendly local game store. Thank you and have fun playing dice with the universe. You.