Is AI actually helpful?

[EmilyMiller]

All of our [interminable] debates about AI seem to assume some sort of benefit to the technology. Arguments instead talk about ethics and loss of human jobs. But what if AI doesn’t actually boost productivity?

This is an excerpt from an article on The Atlantic. It’s behind a paywall (I subscribe), but someone normally finds a free version.

—

If there is any field in which the rise of AI is already said to be rendering humans obsolete—in which the dawn of superintelligence is already upon us—it is coding. This makes the results of a recent study genuinely astonishing.

In the study, published in July, the think tank Model Evaluation & Threat Research randomly assigned a group of experienced software developers to perform coding tasks with or without AI tools. It was the most rigorous test to date of how AI would perform in the real world. Because coding is one of the skills that existing models have largely mastered, just about everyone involved expected AI to generate huge productivity gains. In a pre-experiment survey of experts, the mean prediction was that AI would speed developers’ work by nearly 40 percent. Afterward, the study participants estimated that AI had made them 20 percent faster.

But when the METR team looked at the employees’ actual work output, they found that the developers had completed tasks 20 percent slower when using AI than when working without it. The researchers were stunned. “No one expected that outcome,” Nate Rush, one of the authors of the study, told me. “We didn’t even really consider a slowdown as a possibility.”

No individual experiment should be treated as the final word. But the METR study is, according to many AI experts, the best we have—and it helps make sense of an otherwise paradoxical moment for AI. On the one hand, the United States is undergoing an extraordinary, AI-fueled economic boom: The stock market is soaring thanks to the frothy valuations of AI-associated tech giants, and the real economy is being propelled by hundreds of billions of dollars of spending on data centers and other AI infrastructure. Undergirding all of the investment is the belief that AI will make workers dramatically more productive, which will in turn boost corporate profits to unimaginable levels.

On the other hand, evidence is piling up that AI is failing to deliver in the real world. The tech giants pouring the most money into AI are nowhere close to recouping their investments. Research suggests that the companies trying to incorporate AI have seen virtually no impact on their bottom line. And economists looking for evidence of AI-replaced job displacement have mostly come up empty.

 

[ShelbyDawn57]

From my personal experience, AI has value in stubbing out fundamental functionality, i.e., a wrapper for a micro service based on a class I have already designed. For anything more complex, the cost of refactoring what it creates is too high. I've gotten similar reports from colleagues. In general, where coding is concerned, I feel it creates too much technical debt that a real person then has to go clean up. Technical debt is far more expensive than clean design up front.

When it comes to specific problems with business logic, I don't think AI can have much appreciable value unless the company invests heavily in their own private LLM based on their specific business functionality. I was discussing this with my older brother who is very high up in a financial services company. His take is that, in his world, is that the risks of putting their information into an LLM are just too great. It would make too much protected information public by default. WE were laughing at the risk a certain large multinational bank(WF) was taking by trying it by implementing an Google Cloud solution. Personal information hacks are too common already. Can you imagine having all of your banking history on the cloud, protected by Google? Not me.

 

[TheLobster]

This tracks, and it's not really surprising to anyone more senior in the field.

Having worked in the software engineering industry for well over a decade, I gotta say that the absolute last thing I've ever thought was, "Gee, I sure wish this project wasn't so small! It clearly needs like 100k more lines of code!"

 

[StillStunned]

AI has certainly reduced my workload.

 

[MrPixel]

After 40 years of writing code from BAL to C++ and everything between, my assessment of AI is...

...highly appreciative of the fact that I have been retired for 12 years and it's y'all's problem.

 

[Lumiere_Amie]

I use AI to choose the wine to accompany whatever I'm cooking for dinner.

 

[StillStunned]

I use AI to choose the wine to accompany whatever I'm cooking for dinner.

"Red. One bottle."

 

[AlinaX]

“No one expected that outcome,”

Really? It was tediously predictable, and was predicted by many.

ETA: It’s not that AI can’t be helpful, but AI like so many modern tools is excellent at producing answers that look correct, and that induces laziness in places and wastes time in others.

 

[EmilyMiller]

Really? It was tediously predictable, and was predicted by many.

I’m not on the editorial board of The Atlantic .

 

[BobbyBrandt]

People are trying to lump all uses of artificial intelligence into neat little baskets, typically associated with somehow replacing a human being in some fashion.

I've touched on this in another thread, where I see AI existing in at least two separate and distinct realms; generative and analytical.

At the company I worked at prior to retirement, AI had been implemented as part of the CRM system used for customer and technical support. It increased productivity significantly by offering support personnel an analysis of the customer's symptoms and presenting suggestions on likely solutions from past incidents. Previously, the support personnel would have to manually search through closed cases using keywords to try and align their current incident with the historical data.

These were high-tech, user and device dependent support issues that studies showed could not be handled exclusively by AI, largely because the customers were not generally knowledgeable enough to create queries specific to the symptoms that any AI system could reliably respond to. The support personnel needed to determine the symptoms correctly and then the AI system could provide an analysis of possible solutions that the support personnel could either accept or reject. The system retrieved data, it did not create any.

 

[Lifestyle66]

Really? It was tediously predictable, and was predicted by many.

ETA: It’s not that AI can’t be helpful, but AI like so many modern tools is excellent at producing answers that look correct, and that induces laziness in places and wastes time in others.

Never underestimate the financial rewards from providing even a shitty tool for the lazy to reduce their workload even further!

 

[Robin_Starveling]

Funny how this forum claims to hate AI and LW, yet the chatter about them never stops. Practically spam.

 

[Gamblnluck]

In some ways AI could be a faster solution. Analyzing stocks, anticipating consumer reactions to some trend and drafting a response ahead of time. But is it better?

I was on a cruise and a guy read a couple of my stories. He'd asked why I was on my computer in a common area while I 'people watched'. He thought I was working on a job.

He went on to tell me how I should try 'Chat GTP?" and how it would speed up my writing. I could go back and edit to fit more what I wanted to say. I stopped just short of saying, "I'm writing stories I want to tell. What would be the point? It would not be ME telling those stories."
Even if AI used examples of my writing and my basic idea and started the process, I'd still have to go change it to fit me. I can see how it would even slow down a process.

 

[MrPixel]

I certainly see the value of AI as a programming tool. One of the most tedious tasks when I was coding was writing I/O, particularly user interfaces. It was the same, over and over again, but copying preexisting code never did the whole job, you'd have to re-code whatever was unique to the environment. I can see an AI tool as a means to know, or "learn" the environment(s) so it can do the scutwork.

What I took from the synopsis of the Atlantic article was that it was costing more in time to clean-up problematic code, or trim inefficiencies introduced by one-size-fits-all approaches. That's been the way of the world since...whenever. I could code highly efficient processes in assembler where the generic high-end languages couldn't come close, but it was CPU/processor horsepower that supplanted the need for code efficiency through compact writing. There was an old joke that Windows was a trainload of heavy cars you were trying to pull with a bicycle, but hardware came around to supplying a locomotive, and the woefully-inefficient and cumbersome code didn't need to be pruned as much as you thought it did.

 

[Britva415]

Of course there are also such things as over-investment and failed projects, but there 100% are fields where AI has already impacted productivity immensely, and the areas where potential remains sky-high are numerous.

I'm not an AI fanboy, I'm just informed and know that, as described, the Atlantic article and the research it claims to cite don't pass the smell test.

It's trivially easy to find research which shows how many millions of jobs globally have already been either lost or not created because of the productivity of AI, and it's also trivially easy to find research which shows productivity results which weren't even possible at all without AI. Just for one example, there is NO amount of human capital you could have thrown at, say, protein folding and come remotely close to the results AI yielded.

Were there parameters on the particular fields the Atlantic article was contextualizing? Or the research it cited? Something is missing from the picture here. Not sure if it got lost in the specific research cited, or in the popular media interpreting the research, or in the above summary of the popular-media reporting.

 

[SkyBubble]

All of our [interminable] debates about AI seem to assume some sort of benefit to the technology. Arguments instead talk about ethics and loss of human jobs. But what if AI doesn’t actually boost productivity?

I don't know about productivity, though AI is only going to improve in that regard.

This is a writers' forum. From what I can gather, it's pretty bad at writing. Somebody said it writes like a high school sophomore -- who's getting Cs.

Where I can see writers wanting to use it -- and I'm not recommending it and would not do this myself -- is if you get stuck on where a tory might go. I could imagine a writer who "just can't figure it out" going to Grok or Chat GPT or Gemini and saying I have this story. A, B, and C happen, and I can't figure out whether to do D or E. I understand that temptation.

Teh other thing I can imagine a writer might be tempted to do is go to one or more of those sites saying I have this story. Here's my title. I don't like it. Can you help me think of something better?

Now, I've never done these myself and I believe they're against the rules of this site, but I can understand why someone would be tempted to do so.

 

[EmilyMiller]

though AI is only going to improve in that regard.

This is what the broligarchs say each time their product is shown to be crap. And people keep believing them.

 

[TiberiusPrime]

I use AI to choose the wine to accompany whatever I'm cooking for dinner.

"The wet kind."

 

[Kelliezgirl]

There are areas it's useful, but unlikely that it's this world changing thing.
A friend who is a teacher said it's great for outlining lesson plans. Saves her a few hours a week.
Tech has a long history of over promising and under delivering.

 

[EmilyMiller]

I use AI to choose the wine to accompany whatever I'm cooking for dinner.

The long, drawn out kind, that crescendos until it ends up as a scream, which is then abruptly silenced. All that is left is the sound of crows croaking in raucous delight at the thought of fresh meat.

 

[XerXesXu]

Abstract.

‘Although the influence of experimental artifacts cannot be entirely ruled out, the robustness of the slowdown effect across our analyses suggests it is unlikely to primarily be a function of our experimental design’

Ohh. One of those papers.



Introduction.

‘That said, many of the factors we find evidence for contributing to slowdown are specific to the setting we study—these results do not imply that current AI systems are not useful in many realistic, economically relevant settings. Furthermore, these results do not imply that future models will not speed up developers in this exact setting—this is a salient possibility given the rapid pace of progress in AI capabilities recently [2]. Finally, it remains possible that further improvements to current AI systems (e.g. better prompting/agent scaffolding, or domain-specific finetuning) could yield positive speedup in this setting.’

And only 50 more un-peer–reviewed pages to go. At what point do you ‘nope out’. I've just saved several hours of my life using human intelligence.

I last did any coding back in the 1990’s, when it was first automated. In the intervening three decades, I estimate that automation has saved me approximately 1.8 lifetimes.

 

[ChuckWolf]

Eleven Labs has an AI audio program that is fantastic, however the company can't get out of its own way; the platform is based on credits and requires a lot of trial and error, so you burn up a lot of credits very easily.

More importantly, because of their rigid TOS, the software is not allowed to be used for erotic stories or porn, making it very useless for a site like this.

Also, for a variety of reasons, it's almost impossible to find a site that will allow AI audio to be uploaded there.

Literotica doesn't allow it and I completely get why, for both moral and practical reasons.

 

[Lumiere_Amie]

The long, drawn out kind, that crescendos until it ends up as a scream, which is then abruptly silenced. All that is left is the sound of crows croaking in raucous delight at the thought of fresh meat.

You're thinking whine, not wine, but I like it

 

[EmilyMiller]

You're thinking whine, not wine, but I like it

<deadpan> That had not occurred to me. Thank you for pointing out my schoolgirl error. </deadpan>