"Smart crowdsourcing peer reviews"
Dec. 11th, 2010 12:36 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
walkitoutI feel sorta bad for poking at this phrase, because now I can't stop thinking about it. It is very easy to stop thinking about someone who proposes an R&D group at a University Press -- that's just silly. But "Smart crowdsourcing peer reviews" is tough to let be.
My first thought brought me back to the very mildly interesting Bloomberg news interview with Reid Hoffman that sent me down this whole academic publishing rathole in the first place: maybe LinkedIn could help connect potential, um, manuscripts just seems _really_ wrong in this context, e-books for publication to peer reviewers. Then I got to thinking, today, about the way TUG (Timeshare Users Group -- see, this is what it's like inside my head) solicits reviews from its consumer base.
TUG is a bunch of people who own timeshares and visit timeshares. They are interested in learning about other timeshares, and they know about some timeshares directly. Thus, every participate in the users group is potentially a producer of reviews and a consumer of reviews, and many are both. Same thing in academia. Fresh reviews are better than old reviews (not totally clear how that translates to e-publishing academic books, but hey, nothing is ever exact); more reviews are better than fewer. IIRC, TUG doesn't have any particular reputation system for assessing reviewers, or for assessing the utility of a particular review (Amazon does have the latter; eBay has a parallel for the former).
You could take all this stuff and readily imagine a replacement for editorial staff and/or reviewers in general (such as at Choice), in which you submit a text (has to satisfy format requirements, but you can contract out that part). There could be compensation (honoraria) for the first few reviewers, who might be required to meet particular requirements (registered with the system as having particular credentials in particular areas of expertise). Separate from the "peer review" bar being met, you could also have general (uncompensated, probably) reviews.
It's pretty easy to imagine that some really fast reader with the right credentials in enough fields might decide they could sweatshop themselves and make a living just off of reviews. Whether that's a good thing or not is entirely debatable; some cases might require some investigation to ensure that reviewer ethics are being met. That, in turn, would lead to an entertaining debate on just what reviewer ethics are or should be, exposing a sordid underbelly of academia that could do with a little sunshine. And it's not like we don't have professional readers already, anyway, right?
If you're not asking, why not just do a wiki with credentials, you probably should be.
ETA: Similar ideas used for doing translation work.
http://www.insidefacebook.com/2010/08/09/cloudcrowd-crowdsourcing-peer/
On facebook, no less.
ETAYA: Less sweatshoppy, more relevant to the problem in question:
http://www.nytimes.com/2010/08/24/arts/24peer.html
The commenters were selected to contribute and they signed their work -- and there were other reviewers as well.
The article references this:
http://www.nytimes.com/2010/08/17/science/17proof.htm
How did I miss that!!! I could have had a whole week in August being totally geeky and insufferable following the discussion! Yes, I know. No one else cares.
ETA last time I swear: After digging around in wikipedia for a while and reading about Sokal and the French pseudo-scientist twins and publication subvention and a few other odds and ends, I have concluded that attempting to sweatshop the peer review part of the process is probably missing the point. Totally technically feasible, won't actually help anyone.
My first thought brought me back to the very mildly interesting Bloomberg news interview with Reid Hoffman that sent me down this whole academic publishing rathole in the first place: maybe LinkedIn could help connect potential, um, manuscripts just seems _really_ wrong in this context, e-books for publication to peer reviewers. Then I got to thinking, today, about the way TUG (Timeshare Users Group -- see, this is what it's like inside my head) solicits reviews from its consumer base.
TUG is a bunch of people who own timeshares and visit timeshares. They are interested in learning about other timeshares, and they know about some timeshares directly. Thus, every participate in the users group is potentially a producer of reviews and a consumer of reviews, and many are both. Same thing in academia. Fresh reviews are better than old reviews (not totally clear how that translates to e-publishing academic books, but hey, nothing is ever exact); more reviews are better than fewer. IIRC, TUG doesn't have any particular reputation system for assessing reviewers, or for assessing the utility of a particular review (Amazon does have the latter; eBay has a parallel for the former).
You could take all this stuff and readily imagine a replacement for editorial staff and/or reviewers in general (such as at Choice), in which you submit a text (has to satisfy format requirements, but you can contract out that part). There could be compensation (honoraria) for the first few reviewers, who might be required to meet particular requirements (registered with the system as having particular credentials in particular areas of expertise). Separate from the "peer review" bar being met, you could also have general (uncompensated, probably) reviews.
It's pretty easy to imagine that some really fast reader with the right credentials in enough fields might decide they could sweatshop themselves and make a living just off of reviews. Whether that's a good thing or not is entirely debatable; some cases might require some investigation to ensure that reviewer ethics are being met. That, in turn, would lead to an entertaining debate on just what reviewer ethics are or should be, exposing a sordid underbelly of academia that could do with a little sunshine. And it's not like we don't have professional readers already, anyway, right?
If you're not asking, why not just do a wiki with credentials, you probably should be.
ETA: Similar ideas used for doing translation work.
http://www.insidefacebook.com/2010/08/09/cloudcrowd-crowdsourcing-peer/
On facebook, no less.
ETAYA: Less sweatshoppy, more relevant to the problem in question:
http://www.nytimes.com/2010/08/24/arts/24peer.html
The commenters were selected to contribute and they signed their work -- and there were other reviewers as well.
The article references this:
http://www.nytimes.com/2010/08/17/science/17proof.htm
How did I miss that!!! I could have had a whole week in August being totally geeky and insufferable following the discussion! Yes, I know. No one else cares.
ETA last time I swear: After digging around in wikipedia for a while and reading about Sokal and the French pseudo-scientist twins and publication subvention and a few other odds and ends, I have concluded that attempting to sweatshop the peer review part of the process is probably missing the point. Totally technically feasible, won't actually help anyone.