Ratings Scheme

[alextasy]

I'm of the opinion that any author with fundamental skills and the guts to share her/his/their work on this site deserves all the support we can give them. With one exception, I've given five stars to every story I've read past the first paragraph. The one exception was a hateful diatribe that went nowhere, with no redeeming value except to show the writer could create a loathsome, sociopathic, and hurtful fictional character who despises everyone.

Personally, I don't pay much attention to the ratings when I'm looking through stories. I'm a stickler for grammar, spelling, punctuation, and the basics most of us were taught in elementary school, but I've seen poorly-written stories presented with well-thought-out plots and interesting characters who evolve through their experiences--and having sizzling sex along they way--and I am quick to acknowledge the writer's successes. I've recently learned the "Favorites" numbers don't accurately reflect the actual number of people who have marked a story as a favorite, either, so that tells us nothing about a story's popularity.

Is there no way to revise the ratings scheme to create an environment that fosters and rewards writers and accurately informs readers of the merits of the stories?

The "HOT" label is practically useless. Under the current rating plan, it does not accurately reflect the quality of a story. That is most harmful to the writers, the readers, and the site when a story is in the "NEW" status, when it is most likely to get readers. A story with a rating of 4.8 after nine votes will drop to a 4.3 with a single 'one'-grenade lobbed by some jerk who probably does it for the fun of it, and likely isn't even brave enough to register with the site. At that point, anyone scanning through the listings is less likely to read the story because of the low score. With fewer readers, there are fewer ratings to pull the story back into the statistically inaccurate "HOT" status where more readers will enjoy it.

I restrict comments from anonymous readers. If they don't care to come out from behind their shield of obscurity, then frankly, I don't care to hear what they have to say. Unfortunately, the site does not allow us to restrict ratings from the shadowy masses, which is where I suspect many of those falsely-conceived poor ratings come from.

There are dozens of better ways to inform writers and other readers of the best stories on the site. How about some sort of checkbox feedback that doesn't require writers to submit to scathing personal attacks? For example, for anything below a certain rating (e.g., 3 or 4), the reader (who may not have actually read it) could be required to check the single most important reason for their choice from a list, such as "poor fundamental skills," "disagreeable subject matter," "confusing plot," or similar problems. A single rating like this would not be helpful, but consistent feedback along a single line would give the writer a clue to possible weaknesses.

There has to be a better way. The current method is unnecessarily detrimental to supporting better stories and accurately informing readers.

Thank you for taking the time to consider my criticisms and thoughts on the matter. Not that I expect it will go anywhere, but if we don't express our concerns, then who's to blame?

=Alextasy

 

[alextasy]

Oh, yes...assuming the site uses a standard average to calculate ratings, as my data suggests, in the example of a 4.8 rating dropping to a 4.3 after a single "1" vote, four "5" votes are needed to bring it back into the "HOT" category. Over twenty (20) "5" votes are needed to restore the the rating somewhere near the original 4.8.

A twenty to one ratio... Doesn't seem right.

Thank you.
=Alextasy

 

[Statius]

Oh, yes...assuming the site uses a standard average to calculate ratings, as my data suggests, in the example of a 4.8 rating dropping to a 4.3 after a single "1" vote, four "5" votes are needed to bring it back into the "HOT" category. Over twenty (20) "5" votes are needed to restore the the rating somewhere near the original 4.8.

A twenty to one ratio... Doesn't seem right.

Thank you.
=Alextasy

Agree. I have expressed concerns and doubts about the rating system as well. Prepare yourself for what's coming from other writers though. It probably won't be pretty.

 

[Davidwa1]

Most who give a bad rating are either anonymous or they dont like the subject, or claim not to as they read 10,000 words. I wouldn't give it too much consideration.

 

[alextasy]

Most who give a bad rating are either anonymous or they dont like the subject, or claim not to as they read 10,000 words. I wouldn't give it too much consideration.

It's not me that considers it. It is potential readers who are misled by spurious and possibly spiteful clicks. Just because they might not like the subject doesn't give them carte blanche to dissuade others from enjoying it. I don't like tomatoes. I don't shove my finger into every tomato I see so other people can't eat them.

 

[lovecraft68]

I've been taking flak on this anytime it comes up because I would ditch anonymous in a heartbeat here.

My feeling is even with an account and user handle, how aren't we anonymous? These aren't our names, we're not giving away personal info, and many register with an e-mail they rarely use.

So to still feel the need to be protected even more than that? Come on.

I do know many anon readers vote high and will leave nice comments, but also think if anon suddenly was not an option would they really not create an account?

Because fact is that while most anon folks are fine, 90% of the haters and trolls here are anon and they have proven so cowardly they won't sign up.

What it would come down to is less votes and comments, but better quality and better ratings, which would people prefer?

But the site isn't going to change it so its a moot conversation.

 

[alextasy]

I've been taking flak on this anytime it comes up because I would ditch anonymous in a heartbeat here.

My feeling is even with an account and user handle, how aren't we anonymous? These aren't our names, we're not giving away personal info, and many register with an e-mail they rarely use.

So to still feel the need to be protected even more than that? Come on.

I do know many anon readers vote high and will leave nice comments, but also think if anon suddenly was not an option would they really not create an account?

Because fact is that while most anon folks are fine, 90% of the haters and trolls here are anon and they have proven so cowardly they won't sign up.

What it would come down to is less votes and comments, but better quality and better ratings, which would people prefer?

But the site isn't going to change it so its a moot conversation.

Lovecraft68;

I love your writing. Thanks for your feedback.

I agree anyone can foster hatred. I believe people who have an identity--even if it's a mask with a fake email--are less likely to lob grenades, because they know it can come back on them on other ways. Of course, they can create another identity, but that requires unnecessary effort and a loss of historical data and possibly connectivity with friends, as opposed to simply being civilized.

Again, it's not for my protection. It is a brutish form of censorship by "cowardly" people, a concept this site should be especially sensitive to, and which ultimately affects the readers.

I don't agree the site isn't going to change. It is currently in the process of change. The question is, which direction will it change? I think my concerns are widely shared. I can only hope someone is paying attention to us.

Thank you,
=Alextasy

 

[PennyThompson]

I think we can be pretty confident that Lit's scoring system isn't likely to change, significant modifications to its database system happens rarely and slowly. But it's fun to think about, so here's my modest proposal...

What about a Rotten Tomatoes-style rating system? RT's system is pretty straightforward, it just considers any review of a movie to be either positive or negative.

It doesn't differentiate between a 4-star review and a 5-star review, it just asks, "did the reviewer enjoy it or not?"

And then the aggregate RT score is displayed as a simple percentage of positive reviews. If 60% or more of the reviews are positive, the movie gets a "Fresh" label. Less than 60%, "Rotten."

And then the other interesting thing that RT does that is, it provides two percentage scores. One for professional reviewers and critics, one for public audience comments and scores. This creates a really useful dynamic, I think.

• There are movies that get a low critic score but a high audience score, which usually means it's a dumb popcorn movie that might not tread any interesting artistic ground, but people had a good time watching it.
• There are movies that get a high critic score but a low audience score, which either means it's pretentious oscar bait or unappreciated genius, depending on your personal view of such things
  
• Movies that get high ratings in both scores are almost certainly a worth watching for most people.
• Movies that get low ratings in both scores probably don't have much merit to them, except perhaps as a "so-bad-its-good," movie.

It's not a perfect system, but the math is straightforward while providing a useful amount of nuance.

So in a Literotica RT system, I think you could implement it in two different ways.

1. There is an "Author Rating" and a "Reader Rating." If you're a published author on Lit, any rating you give to a story would be added to the Author Rating. If your account has no published stories, it's a Reader Rating.
2. There's a "Comment Rating" and a "No-Comment Rating." If you spent the time writing a comment, then your rating goes in that score. If you only rate a story, it goes in the other.

Then the rating options would be a simple thumbs-up/thumbs-down rather than a number out of 5. The "Hot" label could be applied to any story that earns 60% (or 70, or 80, or whatever percentage is appropriate) in both scores. Or you could split it into two labels that stories can earn, "Authors Hot" and "Audience Hot." Or you could do away with it entirely, and let the scores speak for themselves!

 

[Baztrachian]

Around Lit I have observed that some people employ the following as their ratings scheme:

• They score every single Loving Wives story as a 1.
• They go out of their way to read NC stories, they are outraged, they announce their outrage, and then score it a 1.
• Every non-erotic story is downvoted for being non-erotic.

Etc.

 

[alextasy]

I think we can be pretty confident that Lit's scoring system isn't likely to change, significant modifications to its database system happens rarely and slowly. But it's fun to think about, so here's my modest proposal...

What about a Rotten Tomatoes-style rating system? RT's system is pretty straightforward, it just considers any review of a movie to be either positive or negative.

It doesn't differentiate between a 4-star review and a 5-star review, it just asks, "did the reviewer enjoy it or not?"

And then the aggregate RT score is displayed as a simple percentage of positive reviews. If 60% or more of the reviews are positive, the movie gets a "Fresh" label. Less than 60%, "Rotten."

And then the other interesting thing that RT does that is, it provides two percentage scores. One for professional reviewers and critics, one for public audience comments and scores. This creates a really useful dynamic, I think.

• There are movies that get a low critic score but a high audience score, which usually means it's a dumb popcorn movie that might not tread any interesting artistic ground, but people had a good time watching it.
• There are movies that get a high critic score but a low audience score, which either means it's pretentious oscar bait or unappreciated genius, depending on your personal view of such things
  
• Movies that get high ratings in both scores are almost certainly a worth watching for most people.
• Movies that get low ratings in both scores probably don't have much merit to them, except perhaps as a "so-bad-its-good," movie.

It's not a perfect system, but the math is straightforward while providing a useful amount of nuance.

So in a Literotica RT system, I think you could implement it in two different ways.

1. There is an "Author Rating" and a "Reader Rating." If you're a published author on Lit, any rating you give to a story would be added to the Author Rating. If your account has no published stories, it's a Reader Rating.
2. There's a "Comment Rating" and a "No-Comment Rating." If you spent the time writing a comment, then your rating goes in that score. If you only rate a story, it goes in the other.

Then the rating options would be a simple thumbs-up/thumbs-down rather than a number out of 5. The "Hot" label could be applied to any story that earns 60% (or 70, or 80, or whatever percentage is appropriate) in both scores. Or you could split it into two labels that stories can earn, "Authors Hot" and "Audience Hot." Or you could do away with it entirely, and let the scores speak for themselves!

That's intriguing, Ms. Thompson. At first look, I like it, especially to show the input from different groups, who would each have a different perspective.

Thank you!
=Alextasy

 

[big_cane_sugar]

to show the writer could create a loathsome, sociopathic, and hurtful fictional character who despises everyone.

Uh oh. That might've been me. If it was, well, at least I did it on purpose!

 

[big_cane_sugar]

I'm happy with the rating system. What might be nice is if readers could make and share lists of their favorite works, similar to playlists on some of our favorite video sites. The reading lists could also be rated.

 

[dream_yahoozie]

I'll admit, I've been a rating watcher. I watched one story slowly climb into the Hot rating after being stuck at a 4.1 for a while.

Is it stupid to think my stories are better when they are liked enough to be rated Hot? Probably.

Do I still like seeing them get that little tag? Yes.

I'll just keep creating what I like, and hoping others like it too, but I agree the ratings system does need a change.

 

[LucieLou007]

Most who give a bad rating are either anonymous or they dont like the subject, or claim not to as they read 10,000 words. I wouldn't give it too much consideration.

That is very true, there are also a lot of trolls who get a kick of giving a 1 star vote.

 

[Biglibido99]

Why not show number of 5 star votes, 4 star votes, 3 star, etc - ala Amazon & other similar sites?
Would that not make visible the average been dragged down by a lone vote?

Also wondering if any authors have looked at the number of people who actually make a rating Vs the number of view?
It's a very low percentage on my best rated story 32700 views, 177 rating, 4.42 score

Maybe if more people actually bothered to rate stories they read, the average wouldn't be so sensitive to the lone bad rating?

 

[darwin1859]

Oh, yes...assuming the site uses a standard average to calculate ratings, as my data suggests, in the example of a 4.8 rating dropping to a 4.3 after a single "1" vote, four "5" votes are needed to bring it back into the "HOT" category. Over twenty (20) "5" votes are needed to restore the the rating somewhere near the original 4.8.

A twenty to one ratio... Doesn't seem right.

You're assuming that the '1' is an error or outlier and calculating the number of '5' ratings needed to "repair" that "error" but that's not the only way to look at it. The mean is a good measure of central tendency (probably the best general-purpose measure). The median also has it's merits is you are worried about "onebombing" (which I think you should be concerned).

In your original post, you talked about preventing anonymous from commenting, but anonymous can still rate your story. You're just preventing them from leaving a written comment. I think it would be better to allow comments and delete the ones you don't like. (Or, not... I don't give a fuck what people write about my stories.)

I think any discussion of improving the rating system is a waste of time, but I don't mind speculating. I like your idea of forcing people to provide a reason, but here are some other thoughts:

1. Force readers to make a written comment and then use AI to derive the numerical rating from those comments (using a custom-tuned ML model). This makes it harder to "onebomb" although I'm sure trolls would find specific comments that trigger a 1.0 and paste them as needed. It also might lead to more extreme comments designed to trigger a 1.0. I guess you could disallow duplicate comments. You could also put a thumbs up/down on the comments and let other people rate the comments. Have the rating system discount the comments with downvotes.

2. Allow readers (perhaps only non-anonymous readers) to leave comments for any part of the story that seems flawed. I've seen websites that do this and it works like Word's comments. This would improve the story more directly and comments would be empirical evidence that the story is less error-free. Not sure how you prevent people from leaving comments that are positive/neutral, or from DOSing this kind of system.

3. Create a "web-of-trust" or "karma" score (through some TBD mechanism; this forum has a "reaction score"). Anonymous users are trusted zero. Incorporate the level of trust in the weighting of the ratings.

4. Similar to your idea, ask reviewers to make more detailed ratings. Not just overall 1-5 but 1-5 on dimensions like "Is the story error free?" "Does the story have interesting characters?" "Does the story have an interesting plot?" This has the advantage that each (logged in) reader could tune their own preferences and have a custom score for how the story would appeal to them. This wouldn't help me a lot, because the nature of the content is probably a lot more influential for me than errors/style.

5. Have AI do the more detailed ratings without the need for humans.

6. Track time spend on the story. Latencies are always difficult to analyze. You don't know if they really spent 2 hours on one page or if they left to have lunch. But if they looked at page 1 for 30 seconds, then skipped to the end and clicked a 1 rating, then they are "onebombing."

7. Find abusive patterns (TBD, but #6 above is an example) and remove those ratings before calculating the mean rating.

8. Have AI review the comments and summarize trends/themes for authors to improve.

No rating system can eliminate bad actors entirely; the best you can do is mitigate harm. All systems balances ease of use with resistance to abuse. The right answer depends on what the community values most.

I asked ChatGPT what I omitted and it added:

9. Community moderation - Letting other users flag suspicious ratings (e.g., a pattern of ones given across many stories by the same person). If multiple people agree it’s malicious, it can be discounted automatically.

10. Gamification and incentives - Reward thoughtful reviewers with badges, visibility, or influence could shift incentives toward better-quality ratings.

11. Statistical safeguards beyond mean/median - Bayesian averaging (shrink averages toward a prior until more ratings accumulate, reducing the impact of early "onebombs.'); Weighted moving averages (newer ratings count more, or trusted raters weigh more heavily)

12. Author-controlled views: Give authors the option to hide ratings altogether and just show comments, or to show only ratings from verified/trusted users.

13. Norm-setting: The system can explicitly encourage constructive ratings: show a nudge like "If you rate this low, please explain why so the author can improve." Sometimes social cues are as effective as technical fixes.

 

[Nynah]

then use AI

Have AI do

Have AI review

I asked ChatGPT

The rating system on Lit isn't perfect, but I would rather keep that than use something that can't find its way out of a paper bag without hallucinating data or manufacturing lies.

 

[darwin1859]

The rating system on Lit isn't perfect, but I would rather keep that than use something that can't find its way out of a paper bag without hallucinating data or manufacturing lies.

Well, the OP seemed to want to discuss alternatives.

And while we are all fully entitled to our opinions, I think it's awfully shorted-sighted to have an attitude of "AI BAD!!!!! AI BAD!!!!! WHERE MY PITCHFORK!!!? WHERE MY TORCH?!?! MUST BURN MONSTER AI!!!!!"

First, the world that didn't have reasonably good AI is gone. We no longer live in that world. And that pre-AI world is never coming back. (You can spit into the wind all you want.) The only change in this area will be gradually better AI.

Second, when you are evaluating the usefulness of AI, the bar is not "Is AI perfect?" but "Is AI practically better than what we have today?" And there are a very, very large class of use cases where we current cannot do anything (because the labor would be ridiculously expensive, the latency would suck, etc.). And for those use cases, it's really easy for AI to be better than nothing. For example, it's currently wildly infeasible for LE staff to police abusive ratings patterns, but AI (or ML) could easily do a reasonable job.

And third, LE already uses AI. All/most of the people complaining "my story was rejected because they said I used AI!!" are probably "false positives" from some ML algorithm that Laurel uses to screen stories. I personally am skeptical of an "AI detector," and that's based on both my knowledge of ML algorithms and statistical decision-making in general, and many reports of false positives. But I'm sure Laurel's POV is that an AI detector is the only practical means of preventing a tidal wave of shitty AI erotica. (And, for the record, I agree that AI-written erotica would be shitty.)

But I think there's no question that Laurel uses "AI" (broadly defined) to help keep this site running smoothly. Do the math: I believe I read Manu say that LE gets something like 400 new stories each day. There's no way Laurel reads 400 stories a day 7 days a week 52 weeks a year. Let's assume Laurel works 8-hour days and reads 250 words/min. She can read about 120,000 words/day. If the typical page/screen is about 3500 words, that's about 34 pages/screens. If the average story has, say, 2 pages/screens, then 400 stories/day would be 800 pages. That's more than 20x more words than Laurel can read in a day. By herself, Laurel can read about 17 stories/day and that's if she spends 8 hours/day 24/7. It doesn't matter if my numbers are off by a bit, three's an order of magnitude mismatch between what one person can do and the volume of the site.

I had once assumed that she had some deputies that read stories for her. I had assumed that each category would have "deputy editors." But you'd probably need a lot of deputy editors and no one (that I've read) in these forums seems to agree that there are a bunch of deputies. And I would assume that those people would be well-known to this community. I have never seen any indication like "You should talk to Soandso about LW stories...". It's always about DMing Laurel. I also find it very hard to believe that Laurel is posting stories without review. So, the only reasonable alternative is that there's some "AI" in the background filtering stories on criteria. Probably there are some keyword filters. Probably in multiple languages.

And we have some evidence of an automated review process. In the forum on rejected stories, there's the suggestions that some automated process analyzes some forms of syntax and rejects stories that fail. The last specific discussion I saw was about how that script would get confused unless each quote was closed (either because of a missing double-quote or because speech was broken into paragraphs). People were getting rejections for this.

So, we seem already have AI/ML/automation to thank for this site continuing to run. And we have evidence that that AI is making errors and yet the world keeps turning. And because AI's already in the building, we'd be foolish to ignore a better application.

A better counter-argument to my AI proposals is that they would cost money and it's not clear that they would improve the site enough to justify any additional cost. (I'm not sure everyone would agree that LE needs to spend any money to fix the rating system).

 

[alohadave]

Well, the OP seemed to want to discuss alternatives.

Yeah, people discuss alternatives to the voting system all the time. It's a recurring theme in the AH. Doesn't mean any of it will or should ever happen.

And while we are all fully entitled to our opinions, I think it's awfully shorted-sighted to have an attitude of "AI BAD!!!!! AI BAD!!!!! WHERE MY PITCHFORK!!!? WHERE MY TORCH?!?! MUST BURN MONSTER AI!!!!!"

First, the world that didn't have reasonably good AI is gone. We no longer live in that world. And that pre-AI world is never coming back. (You can spit into the wind all you want.) The only change in this area will be gradually better AI.

That does not mean that because AI exists, that it must be used.

 

[MaturemanAJ]

This is a good time to ask this question.
I rate stories I read based on how hot the story is. I search titles and I know what I won't bother with.
I will say that stories that have the guy cuming 25 times in 3 hours are just stupid and totally unrealistic.
But let's turn this around and let's say you run a business and you get Google reviews. Sure everyone wants a 5 but people will tell you flat out they don't hand out 5's constantly because nobody is perfect. Then you get the wackos who you will never make happy.
Why would authors here think they are going to get all 5's it makes no sense.
Do they love every movie they watch and rate them a 5 even if the content wasn't that good or the plot faltered?
Remember it's personal likes and dislikes and unless someone takes a personal shot at you who really cares what people think.
I like stories that are personal experiences vs fictional but with that said I don't rate the story poorly because it's fictional

 

[darwin1859]

That does not mean that because AI exists, that it must be used.

Again, we are all fully entitled to our own opinions, but I don't think that's a practical blanket statement because AI is just a technology and useful technologies are pretty addictive.

Substitute any useful technology in that sentence and it seems pretty impractical:

That does not mean that because the Internet exists, that it must be used.
That does not mean that because mobile phones exist, that they must be used.

But maybe I'm misconstruing your point. For example, LE is unlikely to decide to replace authors with AI, even when AI is less shitty at that task. I agree and applaud this policy.

But AI to run the site? Already happening. I think Laurel would say she must use an AI screener, for example. I would be extremely surprised if Laurel and Manu haven't at least investigated using some kind of AI-based nudity screener for images on LE to conform to the new no-nudity policy (to avoid age verification laws), because I cannot image they examined everyone's profile image manually. They deleted my original profile picture of a cartoon with Darwin's head on a chimp's naked body. I'm pretty confident that AI did that; the cartoon was not salacious. I'd be very surprised if Laurel and Manu didn't use some form of the screening tool to separate trans from crossdressing articles when they split that category. And so on...

And I think it's a good thing to use AI to run the site. I doubt LE could exist as it does today without some form of AI/ML/Automation. I'm more concerned that the best AI models are used that produce the fewest errors. (and the least cost/burden so that LE continues to exist).

 

[darwin1859]

This is a good time to ask this question.
I rate stories I read based on how hot the story is. I search titles and I know what I won't bother with.
I will say that stories that have the guy cuming 25 times in 3 hours are just stupid and totally unrealistic.
But let's turn this around and let's say you run a business and you get Google reviews. Sure everyone wants a 5 but people will tell you flat out they don't hand out 5's constantly because nobody is perfect. Then you get the wackos who you will never make happy.
Why would authors here think they are going to get all 5's it makes no sense.
Do they love every movie they watch and rate them a 5 even if the content wasn't that good or the plot faltered?
Remember it's personal likes and dislikes and unless someone takes a personal shot at you who really cares what people think.
I like stories that are personal experiences vs fictional but with that said I don't rate the story poorly because it's fictional

I think some people really care about the ratings and want to fix the perceived flaws in the system. The comments by the OP about "20 5's to balance that 1" clearly indicates a perspective that there are "unfair" 1 ratings.

But it's not shocking that authors want feedback, and one reason to be concerned about ratings is that you might get a handful of comments (or none) and many dozens/hundreds of ratings. It's not ridiculous to want more detailed feedback, especially when the feedback is negative. I think that was the primary point of the OP -- not "Give me all 5's!" but "Tell me why I didn't get a 5 so I can do better"

 

[MaturemanAJ]

I think some people really care about the ratings and want to fix the perceived flaws in the system. The comments by the OP about "20 5's to balance that 1" clearly indicates a perspective that there are "unfair" 1 ratings.

But it's not shocking that authors want feedback, and one reason to be concerned about ratings is that you might get a handful of comments (or none) and many dozens/hundreds of ratings. It's not ridiculous to want more detailed feedback, especially when the feedback is negative. I think that was the primary point of the OP -- not "Give me all 5's!" but "Tell me why I didn't get a 5 so I can do better"

Feedback is great but some feel every story is a 5. I don't think it's a realistic expectation

 

[darwin1859]

Feedback is great but some feel every story is a 5. I don't think it's a realistic expectation

I think the primary point of the OP was not "Give me all 5's!" but "Tell me why I didn't get a 5 so I can do better!" Is that unrealistic?

 

[MaturemanAJ]

I think the primary point of the OP was not "Give me all 5's!" but "Tell me why I didn't get a 5 so I can do better!" Is that unrealistic?

Not at all but " giving all stories a 5 after the first paragraph " is a false positive in my eyes.
All that means is the OP is propping up scores which isn't an accurate ranking system so then why all the complaints about the scoring system?
It goes both ways