Incompetence & Thievery

Hah! I wrote the post title yesterday, reflecting the latest generative AI articles. I came back today & thought "Is this a post about the Orange Turd's administration?" So, correlation or causation? This is definitely the Golden Age of Grifting.

---

My wife used to watch "House" on TV. The ace diagnostician usually was able to get the right diagnosis in only 4-5 tries!

So maybe it's OK that genAI chatbots are wrong 1/2 the time on medical questions. From Gary Marcus:

Please don’t trust your chatbot for medical advice

Four separate studies all point in the same direction

...

All of this – the hallucinations, mistakes, and overconfidence – is entirely typical of LLMs, and entirely problematic in medicine. As the authors put it, in somewhat academic language, but entirely accurately, “continued deployment without public education and oversight risks amplifying misinformation.”

...

In a recurring theme, we see that LLMs don’t know what they don’t know; they work decently well with the information they’ve got but don’t know how to conduct clinical interviews, and in the hands of the lay public can easily give bad advice because the proper questions never get asked, either by the patient or the LLMs. (An expert doctor might use the LLM to better effect, by asking the right questions.)

Yves Smith of "naked capitalism" comments on the 1st study:

Half of AI Health Answers Are Wrong Even Though They Sound Convincing – New Study

It is telling that despite this article describing the abjectly awful performance of AI chatbots that it still gives a ritual defense of their use.

---

Now for the thievery. From 404 media:

This AI Tool Rips Off Open Source Software Without Violating Copyright

Malus, which is a piece of satire but also fully functional, performs a "clean room" clone of open source software, meaning users could then sell software without crediting the original developers.

This is really shitty. Open source software has been responsible for so much advancement. Most projects are kept afloat by a very small number of dedicated individuals, working for the common good & for the love of code.

But Doctorow weighed in on this, saying that it's really not that big a deal, because this "new" software, being AI-produced, cannot be copyrighted.

The (other) problem with automatic conversion of free software to proprietary software:

You can't add ANY license to a public domain work.

---

This last is just sad. From 404 media:

Startups Brag They Spend More Money on AI Than Human Employees

A new class of AI startups say they are taking money that would normally be used to hire people and are spending it on AI compute instead.

...

Startup CEOs who are “tokenmaxxing” are bragging that they are spending more money on AI compute than it would cost to hire human workers. Astronomical AI bills are now, in a certain corner of the tech world, a supposed marker of growth and success.

...

Companies like OpenAI and Anthropic are losing tons of cash on their products; even though artificial intelligence compute is expensive, it is underpriced for what it actually costs, and it’s not clear how long investors in frontier AI companies are going to be willing to subsidize those losses. Meanwhile, we have reported endlessly on “workslop” and the human cleanup that is often needed when AI-written code, AI-generated work, and customer-facing AI products go awry. There are also numerous horror stories of AI getting caught in a loop and burning thousands of dollars worth of tokens on what end up being completely useless tasks. Regardless, there’s an entirely new class of entrepreneur who seems hell-bent on “hiring” AI employees, not human ones.

Of course there's a word for it: "tokenmaxxing". This sure does remind me of other tech bubbles. I still cringe when I hear the word "pivot".

---

Here's the home page/directory for my posts on Bullshit. This is post #118.