Interesting AI story in the Telegraph

[Bramblethorn]

Does it? I believe that question depends on pure logic and nothing else. You cannot differentiate two identical sequences of words, so as long as both the AI and the Human are capable (and likely) to produce the same sequence for whatever reasons, you cannot really differentiate between them.

A lot hangs on the word "likely" there. When we're talking about documents the length of a Literotica story, any given story is extremely unlikely; what really matters is relative likelihood and our priors.

If we sit a monkey down at a typewriter and let it mash keys at random, it's capable of producing a perfect script of Hamlet on its first try, but the likelihood of it doing so is around 1 in 10^350,000. (Numbers are obviously very approximate; based on an estimate of 180,000 characters and a keyboard with 80 possible characters selected with equal probability.)

OTOH, if we take an English speaker who's unfamiliar with Hamlet and ask them to write whatever comes to mind, the likelihood is approximately 1 in 10^54,000. (Based on estimate of 180,000 characters and Shannon entropy of approximately one bit per character for English.)

If I stand outside the room where the monkey and the human are typing, and a manuscript flies out of the window that happens to be a perfect copy of Hamlet, I can estimate the probability that it was written by a human via Bayes' theorem:

P(human author) = L(human would write this)/(L(human would write this) + L(monkey would write this))

i.e. (10^-54000)/(10^-54000 + 10^-350000)

which works out at 99.99999...%, with about 296,000 nines after the decimal point.

So even though both the human and the monkey are capable of writing Hamlet, and both are extremely unlikely to do so, one is much more unlikely to do so, and that lets us be pretty confident that this one was written by a human.

When we're looking at a long sequence of characters (one Literotica page is approximately 20,000 characters), even small differences in the generating mechanism can become detectable, in the same way that even a slightly biased coin can be distinguished from a fair one if one tosses it many times.

Interesting read, but besides the point we are discussing. You are only talking about your own limitation here in expressing some knowledge your brain has acquired. Our brains are tailored to recognize shapes and especially faces that are familiar to us. Our vocabulary is not.

And likewise, our brains are "tailored" (as an atheist, I'd prefer "adapted" here) to deal with languages, to process a string of words and infer things about the speaker/author, without being adapted to articulate how they do that. There doesn't seem to be any obvious reason to assume that a person's inability to describe how they analyse language means they lack the capacity to analyse language.

If somebody is unable to explain to me what their mind and body are doing when they catch a ball or ride a bicycle, do I conclude from that that they're unable to do these things?

An interesting observation, but nothing to do with what you are trying to prove. All you've proven is that words are ill suited to accurately convey certain types of information.

...which is exactly the point.

Now you are talking about a living and evolving language and its rules. I didn't live back in Tolkien's days, but I know for a fact, that just 100 years ago, language was used quite differently. Go back 200 or 300 years and it could start to feel like a foreign language or a very different dialect from what you are used to.

Indeed! And by reading examples of older texts, one can develop the ability to distinguish between material written 200 years ago and material written today, even without being able to write out a comprehensive list of the differences.

I'm not claiming that I or any other human would achieve 100% accuracy on picking AI-generated from human-written, nor that I personally am highly skilled in doing so.

Sooo.. you cannot tell me what makes something feel like it's been written by AI, but you can tell if its written by AI and not someone who is not a native speaker of the language or someone who is too meticulous when it comes to grammar and proper sentence structure.

Excellent reading comprehension. 10/10. That's definitely, absolutely, exactly what I said there.

 

[Bramblethorn]

The plagiarism people need information about WHICH open source material was used. They don't need the training access. They just need to search the inputs. Right?

It's rather more complicated than "just search the inputs".

Suppose I'm writing a scene where a political leader thinks about his supporters. I might write something like this:

He was aware that some among them had suffered greatly. Some had been imprisoned in harsh conditions. Some came from places where they had been victimised for their activism and stunned by the forces of state violence. They were survivors of the hardships that would inspire, and the injustice of their treatment would bless them. They would go back to Jackson, to Montgomery, to Charleston, to Atlanta, to New Orleans, back to the urban jungles of New York and Los Angeles, assured that better times could come, must come to pass.

If I tried to pass that off as my own work, it'd be plagiarism, because I composed that example by making minor alterations to an excerpt from one of Martin Luther King Jr's most famous speeches. Here's the original version:

I am not unmindful that some of you have come here out of great trials and tribulations. Some of you have come fresh from narrow jail cells. Some of you have come from areas where your quest for freedom left you battered by the storms of persecution and staggered by the winds of police brutality. You have been the veterans of creative suffering. Continue to work with the faith that unearned suffering is redemptive. Go back to Mississippi, go back to Alabama, go back to South Carolina, go back to Georgia, go back to Louisiana, go back to the slums and ghettos of our Northern cities, knowing that somehow this situation can and will be changed.

How would you go about teaching a computer to recognise that the former passage is unusually similar to the latter?

 

[Bramblethorn]

Right now, medicine is one of the last places I'd want to apply it to. You appear to have forgotten about AI hallucinations, where it quite happily goes and makes shit up. Not what you want in a diagnosis.

This is one of those times where it really depends on which kind of "AI"/machine-learning we're talking about. LLMs (GPT, Bard etc.) would be completely unsuitable for such purposes, but there are plenty of respectable ML applications doing useful stuff in medicine.

https://www.nature.com/articles/s41591-020-01197-2

https://thevarsity.ca/2023/03/26/what-do-bread-and-cancer-cells-have-in-common/ - I read a more detailed article about this work a while back, but can't locate that one right now.

Machine learning technologies are tools. When applied by people who understand those tools and have a good understanding of the problem to which they're being applied and are willing to put in work to get them working accurately, they can produce great results. But they're also attractive to people who don't want to think about the problem and want a magic robot that will just do all the hard bits for them, and that's where it goes wrong.

The bread-recognition system, for instance - these people spent years getting to understand the problem, they realised that the "one size fits all" DNN approach wasn't appropriate here and figured out a better way to handle this application, and they put a lot of work into getting better input data that the ML system could use more effectively. That's what responsible ML use looks like.

 

[CalebZhass]

A lot hangs on the word "likely" there. When we're talking about documents the length of a Literotica story, any given story is extremely unlikely; what really matters is relative likelihood and our priors.

If we sit a monkey down at a typewriter and let it mash keys at random, it's capable of producing a perfect script of Hamlet on its first try, but the likelihood of it doing so is around 1 in 10^350,000. (Numbers are obviously very approximate; based on an estimate of 180,000 characters and a keyboard with 80 possible characters selected with equal probability.)

OTOH, if we take an English speaker who's unfamiliar with Hamlet and ask them to write whatever comes to mind, the likelihood is approximately 1 in 10^54,000. (Based on estimate of 180,000 characters and Shannon entropy of approximately one bit per character for English.)

Exactly why I used the word "likely". If I were to tell 10 people of similar skill and education to independently give me short to medium length sentences that describe a boy looking at a colorful sunset, I would end up with some pretty similar sentences. Similar education means they have a similar body of work to draw on when describing things. Words are expressions of our thoughts that we learn through example. Either spoken or written. Eventually we come up with expressions of our own (creativity), but in many cases we simply fall back to one of the expressions we heard and liked or a variation of that.

Sure, the laws of probability would dictate, that given enough time, a single monkey hacking away at a typewriter could produce any conceivable work that ever was or will be created by a human. This however would be an unreasonable argument for the simple and obvious time constraint being impossible to meet, not to mention other factors that real life would never have, which the hypothetical scenario would require.

Your whole argument is.. well, I'm not even sure what you are trying to push here. Frankly, you are proving how you cannot compare apples to oranges, pretending that I did, when in fact I didn't. We were not talking about monkeys vs humans. We were talking about machines created to write LIKE HUMANS and humans. Huge, enormous difference.

You can hack away with all the false analogies you want, it will not change the fact, that there are rendered computer graphics, that are so accurate, you would have no way of visually telling them apart from a photograph of a real person for the simple reason that the encoding medium (pixels) is lacking the information depth to allow you to do so. I would propose that even with digital technologies it will get harder and harder to separate deep fakes from real images. This for a medium that's far more complex than words and far more difficult (mechanically) to really capture reality with, than it is to emulate speech or writings that sound human like.

I could give you dozens of examples of paragraph pairs, asking you to pick which one is AI and which one is Human, and I know for a fact, that given a large enough sample your success rate would be more or less 50% as at best you can guess. You can of course say, that it is because you are not skilled in this, but you are missing a point. It is not about skill, it is the similarities of encoded patterns and the practical impossibility to find difference where there is none.

When we're looking at a long sequence of characters (one Literotica page is approximately 20,000 characters), even small differences in the generating mechanism can become detectable, in the same way that even a slightly biased coin can be distinguished from a fair one if one tosses it many times.

You still forget, that what we are talking about here is style.
I asked once already, what are the things we can look for that identifies something as being AI written?
In case of a coin, we know that it should have a 50-50 distribution of the two values and if it does not, its biased.
What are the metrics for text? Sentence length? word repetitions? grammar errors? the lack of grammar errors? lack of colloquialisms? lack of contractions? frequent usage of rare words? mostly only using mundane words?

You can grade a text on those individual metrics, but here is the thing. The score for each category varies from person to person, style to style. What's your criteria for something being an AI? If it falls outside of the norm on all of them, do we then talk surely about something written by an AI? No. It's frigging probabilities. long tail and all that shit. You cannot discount the fact, that for any given mass of average, there will be a substantial mass of outliers as well on the outside thirds of the bell curve.

I see the point you are trying to make, I just don't find it particularly well founded, as you are totally ignoring the complexity of the subject matter being examined and the circumstances of the situation.

And likewise, our brains are "tailored" (as an atheist, I'd prefer "adapted" here) to deal with languages, to process a string of words and infer things about the speaker/author, without being adapted to articulate how they do that. There doesn't seem to be any obvious reason to assume that a person's inability to describe how they analyse language means they lack the capacity to analyse language.

I guess I can accept your argument, that one has to be no cook or understand flavors to be able to tell if a food is good or bad. I am not questioning your ability to tell if a written work is good or bad either. However I maintain, that if you are unable to tell me WHY you think a work is made by AI, then we are not talking about rational thought, but feelings and as such, as much as I respect your feelings, I cannot take them into consideration when I am talking about how we should be dealing with said work.

If you were to talk about something that is more subjective, like style, you can say, that you do not like it. Taste is subjective. Science is not. Justice is not. If you want there to be justice, there has to be rules more defined than: "well, I just think he stole that candy bar".

I'm not claiming that I or any other human would achieve 100% accuracy on picking AI-generated from human-written, nor that I personally am highly skilled in doing so.

Sooo.. you cannot tell me what makes something feel like it's been written by AI, but you can tell if its written by AI and not someone who is not a native speaker of the language or someone who is too meticulous when it comes to grammar and proper sentence structure.

Excellent reading comprehension. 10/10. That's definitely, absolutely, exactly what I said there.

You appear to have missed the line I actually replied to. (under which my quoted text was actually located)

So, no, I reject the idea that inability to succinctly list all the "tells" of AI-generated text disproves the possibility that humans might be able to spot some things that software can't.