Fake positives in AI detection tools, and the fear of being human.

[onehitwanda]

I wonder; Are we okay with Laurel (or other mods) feeding our prose into AI detectors during the vetting process?

Not like we have much of a choice, is it.

 

[SilkyVine]

Not like we have much of a choice, is it.

Short of publishing elsewhere — not really.

https://www.literotica.com/faq/publishing/publishing-ai

"2. Literotica explicitly does NOT grant any person or entity (commercial, non-profit, or other) the legal right to train AI on any works published on Literotica. Each work published on Literotica is copyrighted by the author. Before using any work on Literotica for any purpose (including training AI or any other AI-related use) you are required by law to contact the author to request permission to use that work. Using works on Literotica for training AI without legal authorization may subject you and your AI (and any work generated by your AI) to future lawsuits from the original author(s), Literotica, or both."

This section of their AI policy seems a little redundant if they are in fact, feeding our stories into AI models for them to be trained on. Just food for thought.

 

[AwkwardMD]

This is an understandable, but unfounded, fear

 

[iwatchus]

My speculation is that they do not use an LLM AI checker. Despite the faith some people place in them, I think they are minimally more useful than throwing darts. As some of the disputes over whether a given passage is or is not AI according to the tools om one of these AI threads, maybe this one.

I suspect they do use some simple heuristic checker to flag suspicious writing that Laurel gives a more careful reading to. Because she is still making the ultimate choice, it is alright to have some false positives.

She has scan read something like a million stories by now. When she is in moderation mode, her head has to be in a place that none of us can fully understand. I was in a similar place (different universe) regularly back in my coding days. It's almost like you interact with that universe like you do with the everyday world. You process lots of things and filter them down before you are even consciously aware of them. You have an equivalent of peripheral vision, noticing things you aren't really paying attention to.

I can't imagine she can process the number of stories she does without getting into that zone. And how many stories she has processed, it would be surprising if she did not develop that relationship with the writing.

We do not know exactly how many stories she processes a day, but the stats for ho many are publishes is not hard; @NotWise seems to track it regularly. Recently, it has been reported that about 200 stories a day get published. That does not count any stories that were rejected. Or any edits that are submitted. We know both of those are non-zero. Are we talking 250 reviews a day or 400? I have no idea.

I assume that she recognized red flags (and I assume that the set of red flags continues to evolve) that raise alarm bells about the writing. Manu coded those up and tested them on subsets of stories until they thought they mostly worked. Anything that gets flagged presumably gets more careful attention from Laurel. This is basically the same process I assume for automated checks about age of characters. If something spends more than two or three days in pending without getting approved, it probably tripped some flag. It does not mean she will ultimately reject it. If it spends weeks waiting, it may mean she is torn about accepting it. Or she set it aside and forgot about it, which definitely seems to happen.

I have no inside information; unlike some here, I have never had a meaningful interaction with Laurel (or Manu) and certainly nothing about any of this. But I have built a bunch of software systems and this is how I would imagine it working. YMMV. But cut them some slack, folks. I think this is a labor of love for them. And they are certainly not out to screw any one of us invidually. I know it hurts to be told no. But that's life.

 

[AwkwardMD]

My speculation is that they do not use an LLM AI checker. Despite the faith some people place in them, I think they are minimally more useful than throwing darts. As some of the disputes over whether a given passage is or is not AI according to the tools om one of these AI threads, maybe this one.

I suspect they do use some simple heuristic checker to flag suspicious writing that Laurel gives a more careful reading to. Because she is still making the ultimate choice, it is alright to have some false positives.

She has scan read something like a million stories by now. When she is in moderation mode, her head has to be in a place that none of us can fully understand. I was in a similar place (different universe) regularly back in my coding days. It's almost like you interact with that universe like you do with the everyday world. You process lots of things and filter them down before you are even consciously aware of them. You have an equivalent of peripheral vision, noticing things you aren't really paying attention to.

I can't imagine she can process the number of stories she does without getting into that zone. And how many stories she has processed, it would be surprising if she did not develop that relationship with the writing.

We do not know exactly how many stories she processes a day, but the stats for ho many are publishes is not hard; @NotWise seems to track it regularly. Recently, it has been reported that about 200 stories a day get published. That does not count any stories that were rejected. Or any edits that are submitted. We know both of those are non-zero. Are we talking 250 reviews a day or 400? I have no idea.

I assume that she recognized red flags (and I assume that the set of red flags continues to evolve) that raise alarm bells about the writing. Manu coded those up and tested them on subsets of stories until they thought they mostly worked. Anything that gets flagged presumably gets more careful attention from Laurel. This is basically the same process I assume for automated checks about age of characters. If something spends more than two or three days in pending without getting approved, it probably tripped some flag. It does not mean she will ultimately reject it. If it spends weeks waiting, it may mean she is torn about accepting it. Or she set it aside and forgot about it, which definitely seems to happen.

I have no inside information; unlike some here, I have never had a meaningful interaction with Laurel (or Manu) and certainly nothing about any of this. But I have built a bunch of software systems and this is how I would imagine it working. YMMV. But cut them some slack, folks. I think this is a labor of love for them. And they are certainly not out to screw any one of us invidually. I know it hurts to be told no. But that's life.

 

[AwkwardMD]

We should all, also, take a step back from guessing in a public forum.

 

[SilkyVine]

My speculation is that they do not use an LLM AI checker. Despite the faith some people place in them, I think they are minimally more useful than throwing darts. As some of the disputes over whether a given passage is or is not AI according to the tools om one of these AI threads, maybe this one.

I suspect they do use some simple heuristic checker to flag suspicious writing that Laurel gives a more careful reading to. Because she is still making the ultimate choice, it is alright to have some false positives.

She has scan read something like a million stories by now. When she is in moderation mode, her head has to be in a place that none of us can fully understand. I was in a similar place (different universe) regularly back in my coding days. It's almost like you interact with that universe like you do with the everyday world. You process lots of things and filter them down before you are even consciously aware of them. You have an equivalent of peripheral vision, noticing things you aren't really paying attention to.

I can't imagine she can process the number of stories she does without getting into that zone. And how many stories she has processed, it would be surprising if she did not develop that relationship with the writing.

We do not know exactly how many stories she processes a day, but the stats for ho many are publishes is not hard; @NotWise seems to track it regularly. Recently, it has been reported that about 200 stories a day get published. That does not count any stories that were rejected. Or any edits that are submitted. We know both of those are non-zero. Are we talking 250 reviews a day or 400? I have no idea.

I assume that she recognized red flags (and I assume that the set of red flags continues to evolve) that raise alarm bells about the writing. Manu coded those up and tested them on subsets of stories until they thought they mostly worked. Anything that gets flagged presumably gets more careful attention from Laurel. This is basically the same process I assume for automated checks about age of characters. If something spends more than two or three days in pending without getting approved, it probably tripped some flag. It does not mean she will ultimately reject it. If it spends weeks waiting, it may mean she is torn about accepting it. Or she set it aside and forgot about it, which definitely seems to happen.

I have no inside information; unlike some here, I have never had a meaningful interaction with Laurel (or Manu) and certainly nothing about any of this. But I have built a bunch of software systems and this is how I would imagine it working. YMMV. But cut them some slack, folks. I think this is a labor of love for them. And they are certainly not out to screw any one of us invidually. I know it hurts to be told no. But that's life.

well said.

https://www.literotica.com/faq/publishing/publishing-time

According to the link (as of 2024) they receive roughly 250 submissions per day. Seems about 10 per category go live daily and this is limited on purpose to avoid burying new stories. This is an absolutely huge undertaking for any one person and it is no wonder people are reporting stories taking upwards of two weeks to go live.

 

[jsmiam]

We should all, also, take a step back from guessing in a public forum.

Wait. What? From you?

Have you had conversations with Laurel? I realize that’s a possibility since Laurel chooses to speak through a chosen few rather than directly communicating with the community.

So Yes or no, are you in contact with Laurel and do you have direct information? It’s a fair question.

If you do that’s fine btw. It’s just something you should disclose.

 

[AwkwardMD]

Wait. What? From you?

Have you had conversations with Laurel? I realize that’s a possibility since Laurel chooses to speak through a chosen few rather than directly communicating with the community.

So Yes or no, are you in contact with Laurel and do you have direct information? It’s a fair question.

Over the years I've had almost every kind of rejection Lit has. Did you know Lit used to reject stories if you had a word that is more than 42 characters long? I suspect that specific rejection type went out with the site redesign, but if you were here 5+ years ago you remember the narrow reading column. I have had maybe 15+ rejections, always for different reasons.

In the course of getting those corrected, I have probably had a more-than-nornal amount of communication with Laurel. I wouldn't say it has been a lot of communication, but I believe it’s well above the average.

I have not communicated with her directly about the AI checker. I have communicated with her about it indirectly. I suspect that she wouldn't tell me even if I was right because what I think I know relies on a kind of black box design. It works because nobody understands it. If they did, the checker could be circumvented.

I appreciate that I can be a polarizing personality, and so me putting my thumb on the scale of the AI rejection conversations might lead some people to automatically lean in the other direction, but I can't stop that. I can only try to be helpful.

 

[OverconfidentSarcasm]

AwkwardMD says she thinks it's a homegrown system, not a commercial one like Originality, Sapling, or Copyleaks, though I've heard those mentioned on sites like Reddit as being the system used here.

I'd like to point out that the tools you just mentioned are meant to check for plagiarism, not AI-writing. Those are two VERY different mechanisms.

 

[AwkwardlySet]

My speculation is that they do not use an LLM AI checker. Despite the faith some people place in them, I think they are minimally more useful than throwing darts. As some of the disputes over whether a given passage is or is not AI according to the tools om one of these AI threads, maybe this one.

I suspect they do use some simple heuristic checker to flag suspicious writing that Laurel gives a more careful reading to. Because she is still making the ultimate choice, it is alright to have some false positives.

She has scan read something like a million stories by now. When she is in moderation mode, her head has to be in a place that none of us can fully understand. I was in a similar place (different universe) regularly back in my coding days. It's almost like you interact with that universe like you do with the everyday world. You process lots of things and filter them down before you are even consciously aware of them. You have an equivalent of peripheral vision, noticing things you aren't really paying attention to.

I can't imagine she can process the number of stories she does without getting into that zone. And how many stories she has processed, it would be surprising if she did not develop that relationship with the writing.

We do not know exactly how many stories she processes a day, but the stats for ho many are publishes is not hard; @NotWise seems to track it regularly. Recently, it has been reported that about 200 stories a day get published. That does not count any stories that were rejected. Or any edits that are submitted. We know both of those are non-zero. Are we talking 250 reviews a day or 400? I have no idea.

I assume that she recognized red flags (and I assume that the set of red flags continues to evolve) that raise alarm bells about the writing. Manu coded those up and tested them on subsets of stories until they thought they mostly worked. Anything that gets flagged presumably gets more careful attention from Laurel. This is basically the same process I assume for automated checks about age of characters. If something spends more than two or three days in pending without getting approved, it probably tripped some flag. It does not mean she will ultimately reject it. If it spends weeks waiting, it may mean she is torn about accepting it. Or she set it aside and forgot about it, which definitely seems to happen.

I have no inside information; unlike some here, I have never had a meaningful interaction with Laurel (or Manu) and certainly nothing about any of this. But I have built a bunch of software systems and this is how I would imagine it working. YMMV. But cut them some slack, folks. I think this is a labor of love for them. And they are certainly not out to screw any one of us invidually. I know it hurts to be told no. But that's life.

What a remarkable piece of fan fiction.

 

[OverconfidentSarcasm]

Judging by everything I've seen here, it's highly unlikely that Lit has its own homegrown algorithm for AI detection. An effective algorithm of that type would take an impressive amount of cutting-edge programming skill and knowledge, as well as resources and a lot of time for development and testing. I'm pretty sure that's not the case here.

I'm not so sure about that part, to be honest. I think it could be done rather easily... if you spend some time using LLMs first.

I don't use AI to write my stories. But I do enjoy AI-Chatbots, ever since I learned that I can use them to very easily create text-adventures. I've always been a fan of the Lone Wolf book-series, and with AI-Chatbots, one can just download an LLM, provide it with a text-file that contains the basic world-parameters, and let the AI come up with consequences for the player's choices on the fly.

But, as I played around with this for a while, I quickly noticed that every LLM has its own pet phrases and words, in addition to the general problems all LLMs share. Notable examples of this would be Deepseek's "Not you. Never you", or Claude's "crossing the room in two strides to lean into you", or GPT's "In conclusion". Or simply the damn em-dash.
If you create a list of the most common examples, which can be easily found through Google, and then write a simple script for string comparison, it should make for a pretty good warning system.
(EDIT: I just realized, you probably wouldn't even have to write it yourself. You could just use one of the old SPAM-detection scripts, and make it detect the provided hallmarks instead of words like "Viagra".)

Then you'd actually have to read the story, of course.

LLMs, in general, have a tendency to write uselessly verbose texts that make the whole conversation sound like it's almost semi-poetic. Like, a character can never just be beaten to a bloody pulp, no, their face must be "a mask of crimson and pain". The problem is that LLMs don't understand why the authors whose work they were trained on used those metaphors and hyperboles, so when LLMs do it, it usually ends up sounding empty and not leading anywhere. Or you have the famous "Rule of Three" (not the three laws by Asimov), that they somehow just HAVE to put into their sentences.

The more AI-slop you read, the more obvious it becomes. And, as long as the people who try to submit AI-generated slop don't take the time to understand it themselves, they can not possibly remove all the hallmark signs.

 

[jsmiam]

This is an understandable, but unfounded, fear

The context of this seems to be directed at the post immediately before where you’re saying “no AI checker”. Not entirely clear.

Your “this is it” was more clear. You’re implying no AI checker.

We should all, also, take a step back from guessing in a public forum.

You imply others are guessing.

…

I have not communicated with her directly about the AI checker. I have communicated with her about it indirectly. I suspect that she wouldn't tell me even if I was right because what I think I know relies on a kind of black box design. It works because nobody understands it. If they did, the checker could be circumvented.

The word you won’t use is “yes”. Yes you did yes you have. Just say it for gods sake.

And the black box checker, your words, contradicts the implications you made earlier, steering people from believing there is an AI checker. Both sets of things you said cannot be true at the same time.

So fair question number two: which is it? Pick only one and stick with it please.

 

[StillStunned]

So fair question number two: which is it? Pick only one and stick with it please.

Just because you ask a question, fair or not, doesn't mean you're entitled to an answer.

 

[jsmiam]

I’ll reiterate my own position. I don’t like AI. It is built on theft. Don’t use it. Don’t support it. Scroll past AI summaries when you use internet search. Turn of autocomplete in your word processor and email editor and everywhere else.

It’s not about being backward. It’s about standing up against theft, including of your ideas.

I was involved in AI many years ago, when it basically didn’t work. In earlier times it was based on the pipe-dream of AI products having their own intelligence, not on stealing everything off of the internet. It didn’t work. Now it kinda sorta works in certai ways, but only ways that parlayed its use of training data taken without permission.

 

[jsmiam]

Just because you ask a question, fair or not, doesn't mean you're entitled to an answer.

Was I taking to you?

And when a person is so free flowing with answers on everything, it’s fair to expect a straight answer now and then.

Especially when contradicting themselves.

 

[StillStunned]

Was I taking to you?

You're posting on a public forum. That means you're talking to anyone who cares to reads it.

 

[AwkwardMD]

The context of this seems to be directed at the post immediately before where you’re saying “no AI checker”. Not entirely clear.

No. I was referring to the fear that Lit is feeding every submission into an LLM.

Your “this is it” was more clear. You’re implying no AI checker.

You imply others are guessing.

I am.

The word you won’t use is “yes”. Yes you did yes you have. Just say it for gods sake.

Please have patience with me. It is a difficult subject to have a conversation about. I believe I posses privileged information. Discussing it directly risks exposing things I didn’t mean to expose, and that puts the entire site at risk. I am walking an extremely fine line saying anything at all, but I feel its worth it try to A) help most of us feel comfortable submitting, and B) help the fringe rejections.

And the black box checker, your words, contradicts the implications you made earlier, steering people from believing there is an AI checker. Both sets of things you said cannot be true at the same time.

So fair question number two: which is it? Pick only one and stick with it please.

This is a conflation of terms that I am guilty of. Yes, I believe Lit has a system it uses to detect AI. No, I believe it is not GPTZero, or any of the LLM-derived services commonly described as "AI Detectors". I have tried to be consistent, but I am constantly bumping up against elements of Lit's system that I know I can't explain and that makes it hard to grapple with or have a direct conversation around.

I first discovered what I discovered in June of 2024. I talked to several AH members about a month later in private (SimonDoom, EB, and AwkwardlySet) in a kind of panic because I didn't know how to talk about it all but felt compelled to all the same, for the same reasons listed above. There's a lot of confusion around the subject, a lot of understandable fear, and I am trying to help allay that.

I am not the perfect spokeswoman, but I am doing my best.

 

[jsmiam]

You're posting on a public forum. That means you're talking to anyone who cares to reads it.

And aren’t you stealing someone’s job?

Omen is the one supposed to rush to defend Awk, just as Awk is supposed to be the one to rush to defend omen.

You sir, are interloping!

 

[jsmiam]

No. I was referring to the fear that Lit is feeding every submission into an LLM.


I am.



Please have patience with me. It is a difficult subject to have a conversation about. I believe I posses privileged information. Discussing it directly risks exposing things I didn’t mean to expose, and that puts the entire site at risk. I am walking an extremely fine line saying anything at all, but I feel its worth it try to A) help most of us feel comfortable submitting, and B) help the fringe rejections.


This is a conflation of terms that I am guilty of. Yes, I believe Lit has a system it uses to detect AI. No, I believe it is not GPTZero, or any of the LLM-derived services commonly described as "AI Detectors". I have tried to be consistent, but I am constantly bumping up against elements of Lit's system that I know I can't explain and that makes it hard to grapple with or have a direct conversation around.

I first discovered what I discovered in June of 2024. I talked to several AH members about a month later in private (SimonDoom, EB, and AwkwardlySet) in a kind of panic because I didn't know how to talk about it all but felt compelled to all the same, for the same reasons listed above. There's a lot of confusion around the subject, a lot of understandable fear, and I am trying to help allay that.

I am not the perfect spokeswoman, but I am doing my best.

I appreciate this reply. Nobody is asking for revealing secrets, but just acknowledging you have some secret information is progress.

 

[AwkwardMD]

For example, I try to use the term "Lit's AI Detector", and for me the air quotes are around all three words. I can completely understand someone else reading that and only putting them around "AI Detector", and thinking that I'm being hypocritical or clever or unclear.

EDIT: My use of the term "Lit's AI Detector" is a result of my evolving attempts to discuss something that is hard to discuss. I haven't always used this term because I didn't realize I needed to before a certain point, and I haven't been perfect using it since then because I'm human.

 

[AwkwardMD]

All that being said, guessing and comparing notes on what does and doesn't get rejected helps no one here in the AH but potentially enables those with less-than-generous motives

Write your own stories. Edit your own stories (or enlist other people to help you with that). You won't have a problem.

 

[AwkwardMD]

I’ll reiterate my own position. I don’t like AI. It is built on theft. Don’t use it. Don’t support it. Scroll past AI summaries when you use internet search. Turn of autocomplete in your word processor and email editor and everywhere else.

It’s not about being backward. It’s about standing up against theft, including of your ideas.

I was involved in AI many years ago, when it basically didn’t work. In earlier times it was based on the pipe-dream of AI products having their own intelligence, not on stealing everything off of the internet. It didn’t work. Now it kinda sorta works in certai ways, but only ways that parlayed its use of training data taken without permission.

This is the way

 

[AG31]

I tried to reproduce your report that 1984 was qualified as AI. I took three different segments of the text from the Gutenburg copy and put them through three different detectors. They all registered as 100% human.

I tried with "For Whom the Bell Tolls," also using the Gutenburg text. I used three different sections of text from different parts of the story and put them through ZeroGPT. The results were 100% human in every case. Ernest would probably curse me for checking.

@Matt4224, which pieces of those novels did you use?

 

[AG31]

I recently had a story rejected for using AI. It's a false positive: ie I didn't use AI.

However, it's a bit hard to suggest one isn't using an app that has AI assistance these days. Word has CoPilot, Apple Pages has Apple Intelligence, and Grammarly is not uncommon. For years, many writing apps have been offering simple grammatical assistance for better sentence structure.

I worry about assistance-creep (my way to describe it). Now with AI so ingrained, does that make suggestions a risk? I think if it just addresses the sentence structure, like spelling, or missed punctuation, that seems ok. But what if it suggests your sentence is too informal? Or to sound more casual? Seems like a grey area to me, because the idea is still my own, but the influence may be impactful. Does that make it collaborative? This is where my discomfort lies.

BTW, when I got the rejection recently, I oscillated between "do you think it is that well written?," and, "wait, is my writing only as good as some AI?" I guess I didn't know what to think, except I didn't use one.

You don't have to turn on those assistants.