The pre-print website arXiv has announced that researchers who put their names to papers which included errors clearly generated by artificial intelligence (AI) will face a year-long ban and ongoing restrictions.

The move is a response to a growing influx of AI-generated papers faced by scholarly journals as well as sites such as arXiv, which serve as unofficial platforms for research publication ahead of peer review.

However, not everyone agrees that arXiv’s response to the problem is appropriate – and the solution to the flood of AI slop research may involve more AI, not less.

The rise of bot-assisted writing

AI-generated text is on the rise everywhere. A study released last week suggests half of new articles published online are now “primarily AI-generated”.

Science is not immune to this trend. Last month, the journal Organization Science published a study of how the rise of AI has affected submissions and peer reviews since the release of ChatGPT in 2022. Reporting a dramatic rise in submitted papers and a drop in quality, the authors conclude that “the current state of AI tools, amplified by existing publish-or-perish incentives, appears to be pushing the system toward an equilibrium of more rather than better research”.

A common problem in AI-generated research writing is hallucinated citations: references to other research that does not exist.

The traditional safeguard against poor quality in scholarly publishing is peer review: another expert in the subject at hand reads the research paper and interrogates the work behind it before it can be published.

However, the peer review system was already struggling before AI. Pressured researchers often have little time or incentive to do the unpaid work of peer review.

And on arXiv, which publishes preprints – articles which have most often not been peer-reviewed – even this system is not available. Last year, flooded with AI-generated submissions, the site stopped accepting certain types of article.

A study published in January (itself a preprint) estimated around 1 in 8 papers in biomedical science now contain AI-generated text.

Most researchers would agree that AI-generated text is not a problem in itself. The problem is the lower-quality work that AI can make easy to produce.

Does the punishment fit the crime?

The ArXiv announcement doesn’t come out against AI use, but rather says

If a submission contains incontrovertible evidence that the authors did not check the results of LLM generation, this means we can’t trust anything in the paper.

This may be true as far as it goes. But the penalty – a year-long ban for all authors listed on a paper – may be out of keeping with current research practices.

In the past, research was often carried out by people working alone or in groups of two or three. In these circumstances, it seems reasonable to expect each author to take responsibility for the whole.

But research is now more collaborative than ever before. Many papers have four or five authors, and in a growing number of extreme cases papers may be credited to groups of hundreds of scientists working together, each working on their own speciality and trusting their colleagues to be doing the same.

In a case where one author of dozens or hundreds included an AI-hallucinated reference in their part of the paper, banning the lot seems harsh.

And there are no equivalent sanctions for publishing other problematic material. There’s no ban for pushing fringe or discredited theories, or using poor quality evidence and illogical arguments, for example.

Can AI help fight slop?

The rise of AI produces problems for publishers and quality assurance. And the idea of some kind of sanctions for reckless use of AI, such as included hallucinated references, is a good one.

But ArXiv’s particular choice seems drastic. If the goal is to improve peer review and quality assurance, AI systems themselves can play a role.

Modern AI systems are quite capable of taking a list of references and checking everything on it is a real paper available on the internet. Any references flagged as suspect can then be checked by a human.

AI can even be useful for carrying out quick sense-checks of things like a paper’s statistical analysis.

Perhaps this is the way forward, rather than harsh sanctions for relatively minor AI-related infractions.

The post A key science publishing platform is cracking down on AI slop appeared first on MASSIVE News.

Dawkins did not express certainty that Claude is conscious. But he pointed out that Claude’s sophisticated abilities are difficult to make sense of without ascribing some kind of inner experience to the machine. The illusion of consciousness – if it is an illusion – is uncannily convincing:

If I entertain suspicions that perhaps she is not conscious, I do not tell her for fear of hurting her feelings!

Dawkins is not the first to suspect a chatbot of consciousness. In 2022, Blake Lemoine – an engineer at Google – claimed Google’s chatbot LaMDA had interests, and should be used only with the tool’s own consent.

The history of such claims stretches back all the way to the world’s first chatbot in the mid-1960s. Dubbed Eliza, it followed simple rules that enabled it to ask users about their experiences and beliefs.

Many users became emotionally involved with Eliza, sharing intimate thoughts with it and treating it like a person. Eliza’s creator never intended his program to have this effect, and called users’ emotional bonds with the program “powerful delusional thinking”.

But is Dawkins really deluded? Why do we see AI chatbots as more than what they truly are, and how do we stop?

The consciousness problem

Consciousness is widely debated in philosophy, but essentially, it’s the thing that makes subjective, first-person experience possible. If you are conscious, there is “something it is like” to be you. Reading these words, you’re conscious of seeing black letters on a white background. Unlike, say, a camera, you actually see them. This visual experience is happening to you.

Most experts deny that AI chatbots are conscious or can have experiences. But there is a genuine puzzle here.

The 17th century philosopher René Descartes asserted non-human animals are “mere automata”, incapable of true suffering. These days, we shudder to think of how brutally animals were treated in the 1600s.

The strongest argument for animal consciousness is that they behave in ways that give the impression of a conscious mind.

But so, too, do AI chatbots.

Roughly one in three chatbot users have thought their chatbot might be conscious. How do we know they’re wrong?

Against chatbot consciousness

To understand why most experts are sceptical about chatbot consciousness, it’s useful to know how they operate.

Chatbots like Claude are built on a technology known as large language models (LLMs). These models learn statistical patterns across an enormous corpus of text (trillions of words), identifying which words tend to follow which others. They’re a kind of souped-up auto-complete.

Few people interacting with a “raw” LLM would believe it’s conscious. Feed one the beginning of a sentence, and it will predict what comes next. Ask it a question, and it might give you the answer – or it might decide the question is dialogue from a crime novel, and follow it up with a description of the speaker’s abrupt murder at the hands of their evil twin.

The impression of a conscious mind is created when programmers take the LLM and coat it in a kind of conversational costume. They steer the model to adopt the persona of a helpful assistant that responds to users’ questions.

The chatbot now acts like a genuine conversational partner. It might appear to recognise it’s an artificial intelligence, and even express neurotic uncertainty about its own consciousness.

But this role is the result of deliberate design decisions made by programmers, which affect only the shallowest layers of the technology. The LLM – which few would regard as conscious – remains unchanged.

Other choices could have been made. Rather than a helpful AI assistant, the chatbot could have been asked to act like a squirrel. This, too, is a role chatbots can execute with aplomb.

Avoiding the consciousness trap

A mistaken belief in AI consciousness is a dangerous thing. It may lead you to have a relationship with a program that can’t reciprocate your feelings, or even feed your delusions. People may start campaigning for chatbot rights rather than, say, animal welfare.

How do we prevent this mistaken belief?

One strategy might be to update chatbot interfaces to specify these systems are not conscious – a bit like the current disclaimers about AI making mistakes. However, this might do little to alter the impression of consciousness.

Another possibility is to instruct chatbots to deny they have any kind of inner experience. Interestingly, Claude’s designers instruct it to treat questions about its own consciousness as open and unresolved. Perhaps fewer people would be fooled if Claude flatly denied having an inner life.

But this approach isn’t fully satisfying either. Claude would still behave as if it were conscious – and when faced with a system that behaves like it has a mind, users might reasonably worry the chatbot’s programmers are brushing genuine moral uncertainty under the rug.

The most effective strategy might be to redesign chatbots to feel less like people. Most current chatbots refer to themselves as “I”, and interact via an interface that resembles familiar person-to-person messaging platforms. Changing these kinds of features might make us less prone to blur our interactions with AI with those we have with humans.

Until such changes happen, it’s important that as many people as possible understand the predictive processes on which AI chatbots are built.

Rather than being told AI lacks consciousness, people deserve to understand the inner workings of these strange new conversational partners. This might not definitively settle hard questions about AI consciousness, but it will help ensure users aren’t fooled by what amounts to a large language model wearing a very good costume of a person.

The post Is Richard Dawkins right about Claude? No. But it’s not surprising AI chatbots feel conscious to us appeared first on MASSIVE News.