Strange Rankings

This is neither good or bad, just odd.

When the first chapter of the story I'm working on was published I noticed that it was up some hours later. It had a 4.88 score at the time, which I knew was ridiculous.

It's hard for me to be objective but I thought it deserved something in the 4.3 range and my goal was a 4.5. The score slowly slid down and settled at around a 4.65.

Chapter two was released and I again realized it hours later. When I took a look it was sitting at a 4.31. It slowly rose and is now at a 4.64.

Almost identical scores coming from opposite directions. Strange.

i don't know for sure but i think there are trolls who go through the new stories and deliberately give a low rating to anything they see doing well. also, i've noticed that the ratings are completely different in different areas. something that does well in one area will do poorly if the next chapter is posted under a different subject. i've given up on the whole rating system. if i like what i've posted, then fuck'em if they can't take a joke.

It's nearly impossible to infer anything meaningful from early scores, because since there are so few of them, outliers, either good or bad, will affect the number significantly. You have to wait until the rating is an average of a meaningful number of votes. Then it begins to mean something.

Variance depend on sample size. Early on, when you have few votes, the current score will deviate a lot from it's true mean (which you would get with infinite number of votes). With more votes, the score converges.

tomlitilia said:

Variance depend on sample size. Early on, when you have few votes, the current score will deviate a lot from it's true mean (which you would get with infinite number of votes). With more votes, the score converges.

Click to expand...

It's interesting, after a while, how stable scores are. I've noticed that when the troll-bashing is done, usually within the first few days of publication, scores tend to rise a little, but then they stabilize, and after a month or two there's very little change even when stories continue to get votes.

................

KindofHere said:

I don't know if my latest will rise, but it got to 4.79 and then a 1 vote dropped it to 4.78, and then it got to 4.79, and a few more votes kept it there, then a 1 dropped it to 4.78, and this continued for a while, 4.79, to 4.78, and then to 4.77 and it fought with 4.78, and now whoever was doing it seems satisfied that it's steady at 4.77. I was able to keep track because I was making notes on my next story on my computer with my author page up. I'm sure after a sweep it will go up, but it's annoying to see it being done so deliberately right in front of my eyes. I know it was deliberate, because all votes were 5s until it gained that .01, and then the low vote came either right away or within twenty minutes.

Click to expand...

I can sympathize because I sometimes track scores with the same sort of obsessiveness, but in my saner moments I realize there's something ridiculous about caring about whether a story has a 4.79 or 4.78.

Getting to 4.5 matters because it makes stories more visible in searches. A high score can get one on or near the top of a toplist, which is helpful if one's goal is to reach readers (mine is).

But once it's sufficiently over 4.5 and safely ensconced there, minor fluctuations don't matter much.

SimonDoom said:

It's interesting, after a while, how stable scores are. I've noticed that when the troll-bashing is done, usually within the first few days of publication, scores tend to rise a little, but then they stabilize, and after a month or two there's very little change even when stories continue to get votes.

Click to expand...

Agree this. Pay attention a month in, is my advice. That's time for the score numbers to settle, one or two sweeps will have gone through (all stories get swept, not just contest entries). I've found my scores slowly strengthen over time - I put this down to a slowly widening fan base who discover my writing today, and slowly wander through my back catalogue.

For interest, I've attached a couple of graphs for six stories (names and details withheld) showing how the ratings changed with increasing number of votes. The first graph shows the ratings out to 100 votes; the second graph shows ratings to 1000 votes, with votes on a log scale. (Edit: my mistake, 300 votes not 1000; not sure how that snuck in.)

I'd be interested in people's opinions on the patterns in these graphs - do any of these look like systematic bombing or boosting to you? I'll give a bit more information about these later, but for now I'd like to know what people make of these based solely on the data shown.

..................

Bramblethorn said:

For interest, I've attached a couple of graphs for six stories (names and details withheld) showing how the ratings changed with increasing number of votes. The first graph shows the ratings out to 100 votes; the second graph shows ratings to 1000 votes, with votes on a log scale.

I'd be interested in people's opinions on the patterns in these graphs - do any of these look like systematic bombing or boosting to you? I'll give a bit more information about these later, but for now I'd like to know what people make of these based solely on the data shown.

Click to expand...

Clear trends for each story as I would expect to see; except for Story 2, which takes the dip and then trends steadily upwards. That's the one that intrigues me, purely because it shows noticeably different reader behaviours over time.

Bramblethorn said:

For interest, I've attached a couple of graphs for six stories (names and details withheld) showing how the ratings changed with increasing number of votes. The first graph shows the ratings out to 100 votes; the second graph shows ratings to 1000 votes, with votes on a log scale.

I'd be interested in people's opinions on the patterns in these graphs - do any of these look like systematic bombing or boosting to you? I'll give a bit more information about these later, but for now I'd like to know what people make of these based solely on the data shown.

Click to expand...

What I see is that some start low (the common early 1* bomb, I imagine) and work up and while others start high and work down. I think it's odd that all of the stories but Story 1 seem to stabilize, but Story 1 gradually sinks.

I plotted up some of mine in the same format and the biggest pattern difference I see is that my graphs show a lot of sweeps, which create a very jagged effect.

In early voting your curves commonly show drops followed by a series of 5* votes that create rising, concave-downward curves. I don't see that kind of regular pattern in early voting.

Systematic bombing? Maybe Story 1 was systematically up-voted in the the early going and after that stopped it settled back.

I did see a possibly systematic pattern on a recent story. It started low then after the initial flurry of votes pushed it over 4.5 it was knocked down below 4.5. Each time it exceeded 4.5 again it was down-voted. That felt pretty systematic to me, but it's impossible to say whether that's one person or just a common behavior in the hub (EC). It climbed just over 4.5 and stayed there after the story dropped off the hub.

Now, compare the top half of your story file to the bottom half. That's where you will find the people you've pissed off in the threads. They usually bomb down until they lose their mad.

Another thing to figure in on the drops is your position on the top lists and who is above you. Fans do strange things to keep their authors on top.

Bramblethorn said:

For interest, I've attached a couple of graphs for six stories (names and details withheld) showing how the ratings changed with increasing number of votes. The first graph shows the ratings out to 100 votes; the second graph shows ratings to 1000 votes, with votes on a log scale.

I'd be interested in people's opinions on the patterns in these graphs - do any of these look like systematic bombing or boosting to you? I'll give a bit more information about these later, but for now I'd like to know what people make of these based solely on the data shown.

Click to expand...

I don't see enough information here to make an inference about bombing or boosting, but the thing that strikes me is that there appears to be a regression by your stories, collectively, to an author mean. Your high story comes down over time. Your low story goes up. Over time, the gap between high story and low story decreases, just a bit, and stabilizes.

It would be interesting to see similar graphs for other authors. I haven't done such a graph for my stories, but I think the graphic result would be somewhat similar. My highest-rated story peaked at a score higher than it is today, and my lowest rated stories were rated lower in the past than they are today.

Thanks all for your comments on those graphs. Time for a confession on my part: those are not from actual Literotica stories. I simulated them in an Excel spreadsheet using a random number generator.

For stories #1-3, the mechanism is simply a coin-toss: each vote has a 50% chance of being a 4, and a 50% chance of being a 5.

For stories #4-6, each vote has a 20% chance of being a 4 and an 80% chance of being a 5.

There are no 1s, 2s, or 3s. There are no systematic effects over time - every vote is generated by exactly the same mechanism as the one before it, and independent of previous outcomes. All the patterns you've noticed in those graphs are just the result of that random process.

electricblue66 said:

Clear trends for each story as I would expect to see; except for Story 2, which takes the dip and then trends steadily upwards. That's the one that intrigues me, purely because it shows noticeably different reader behaviours over time.

Click to expand...

The semblance of long-term trends is basically because the early scores often contain a lot of "error". If you toss a coin 50 times, the average result is exactly 50% heads. But there's only about a 10% probability of getting that exact 50-50 result; 90% of the time you'll end up with something higher or lower, possibly quite a lot higher/lower.

Over time, it averages out, so the score drifts towards the long-term average. In this simulation, Story 1 lucked out in the early runs - even though its true "quality" is only 4.5, it got more than the expected number of 5s, and even up to ~ 50 votes it was sitting near 4.7. (Meanwhile, Story 6, which is really a 4.8, got a bit unlucky and was sitting on 4.7).

In the long run, though, those lucky/unlucky streaks ended, and the scores drifted slowly and erratically towards their long-term averages. If #4-6 look like they're trending upwards over time, it's just because their initial scores were a little bit low and the long-term averaging is correcting that.

(But, human psychology being what it is, we put a bit too much stock in the early, imprecise information, and start looking for explanations as to why the numbers are changing, rather than asking how trustworthy they were in the first place.)

Story 2 actually settles near its long-term average quite quickly, but then it had an unlucky streak around votes 45-55 followed by a bit of good luck up to about vote 100.

NotWise said:

What I see is that some start low (the common early 1* bomb, I imagine) and work up and while others start high and work down. I think it's odd that all of the stories but Story 1 seem to stabilize, but Story 1 gradually sinks.

I plotted up some of mine in the same format and the biggest pattern difference I see is that my graphs show a lot of sweeps, which create a very jagged effect.

In early voting your curves commonly show drops followed by a series of 5* votes that create rising, concave-downward curves. I don't see that kind of regular pattern in early voting.

Systematic bombing? Maybe Story 1 was systematically up-voted in the the early going and after that stopped it settled back.

Click to expand...

Story 1 is indeed stabilising; it's just that this kind of stabilisation takes longer than intuition tends to expect.

That sequence of a gradual, concave rise interrupted by abrupt drops is pretty much what you'd expect for a story where most votes are 5s and the rest are 4s. When the score is in the neighbourhood of 4.8, one 4 cancels out all the good work of the last four 5s; you're likely to get a run of 5s that each bring the score up a little bit, and then one 4 that drops it back down again.

For a story around 4.5, the behaviour should look a bit more symmetrical, and that's pretty much what happens here, at least with #2 and #3, with uphills and downhills having the same sort of length and slope.

SimonDoom said:

I don't see enough information here to make an inference about bombing or boosting, but the thing that strikes me is that there appears to be a regression by your stories, collectively, to an author mean. Your high story comes down over time. Your low story goes up. Over time, the gap between high story and low story decreases, just a bit, and stabilizes.

Click to expand...

Yep. Regression to the mean is exactly right. Story #1 is a good illustration of this: it's not that anybody's targeting it for downvoting, just that it got lucky early on and got a higher rating than it really deserved, and then over time the effects of that luck became less important and it drifted back towards where it ought to be.

It would be interesting to see similar graphs for other authors. I haven't done such a graph for my stories, but I think the graphic result would be somewhat similar. My highest-rated story peaked at a score higher than it is today, and my lowest rated stories were rated lower in the past than they are today.

Click to expand...

Indeed, and this will tend to happen with just about any randomly-generated number series of this type. The stories that rate highest will probably have that status because of a combination of quality and luck, and over time the luck goes away so they'll drop down a bit.

I'm certainly not discounting the existence of things like 1-bombing or long-term trends in story scores, but I think people are a little too quick to assign meaning to the patterns they see. Up to about 100 votes, scores can be significantly influenced by pure chance.

In the example I posted, there's a point out around 55 votes where Story 1 is averaging ~ 4.7 and Story 2 is averaging 4.4. Those two stories are exactly the same in quality, headed for exactly the same long-term average when they get enough votes; the difference is entirely due to the luck of the RNG. But if they were real stories by the same author, that author would probably be coming here or Story Feedback asking people to read them both and figure out why one scored so much better than the other.

Bramblethorn said:

Thanks all for your comments on those graphs. Time for a confession on my part: those are not from actual Literotica stories. I simulated them in an Excel spreadsheet using a random number generator.

Click to expand...

Very clever! Now I'm curious whether the results of real stories track these random graphs or whether they deviate in nonrandom ways.

SimonDoom said:

Very clever! Now I'm curious whether the results of real stories track these random graphs or whether they deviate in nonrandom ways.

Click to expand...

Here are two more sets.

The first graph shows five late chapters from a single story. Every one of the chapters starts low and the score increases with time. Sweeps are a noticeable factor.



The second graph shows five standalone stories ranging from a few months old to a year and a half old. I think there may be two non-random patterns in the graph. Story 4 was posted to a category where it wasn't very popular. Initial votes were pretty good but the category readers seemed to batter it down. The score has increased since it slipped off the hub. Story 5 initially rose above 4.5 in a flurry of votes then was bombed back. It isn't entirely obvious on the graph, but from then until it slipped off the category hub it was bombed back as soon as it reached 4.5--usually on the next vote. It has gone over 4.5 since disappearing from the hub.

SimonDoom said:

Very clever! Now I'm curious whether the results of real stories track these random graphs or whether they deviate in nonrandom ways.

Click to expand...

Votes vs. score for my stories attached.

Note - I didn't record every individual vote, these are from snapshots taken anywhere from hours to months apart (more often when I had a new story up). Rounding makes these plots a bit more jerky than the simulated ones I posted above.

General observations: as expected, up to ~ 100 votes the scores are all over the place, with some big declines and increases depending on where the rating started.

After that, I think it's fair to say my big series (thin blue, purple, red lines) shows a bit of an increase over time. That makes sense - the people who read it early on might just be browsing anything from New Stories, but the ones who find it later probably found it via a more tailored search, or by reading one of my stories and then looking for more, so it's likely to be closer to their tastes.

Of the rest: three haven't broken 100 yet (Erotic Horror and Text with Audio don't seem to be heavily read) and another has only just done so. The two "Red Scarf" stories seem to be on an increasing trend (n.b. these are my most recent, posted on consecutive days, so those views would almost all be coming from New Stories/Category Hub). On the other hand, "Counting to Eleven" seems to be more or less holding level; if there is an overall upwards or downwards trend there, it's masked by random noise.

While setting that up, I accidentally created a plot of votes vs. views (also attached, with closeup), which was a bit more interesting than I'd expected. Please note that views are on the y-axis, so steeper slope = smaller % of viewers going on to vote.

The striking thing in this one is that the ratio changes. The data's a bit noisy, but it looks as if almost all of them show an upwards bend, i.e. the early viewers are more likely to vote than later ones.

Question to those of you with the graphs: how are you creating them? What app?

SimonDoom said:

Question to those of you with the graphs: how are you creating them? What app?

Click to expand...

LibreOffice Calc (a spreadsheet program) in my case. It's free.

SimonDoom said:

Question to those of you with the graphs: how are you creating them? What app?

Click to expand...

I'm using Excel, though as the amount of data increases I'm starting to think about switching over to something more easily programmable.