I Thought Prompt Injection Was A Vaccine Thing Until I Discovered AI

Information Week

The generative AI hype bubble definitely deflated significantly during the summer. Gartner calls it the "trough of disillusionment" in the "AI hype-cycle" and points to 'agentic AI' and 'AI-native software engineering' (aka 'vibe coding') as (somewhat) distinct new entrants. Why the disillusionment? Well, on top of earlier risk management warnings, we've heard lots about the fact that inaccuracy, bias and hallucination are features of generative AI, rather than bugs, as explained very well in the 'myth busting' post by The Guardian. But what we're really hearing about now are the security vulnerabilities, which seem even more problematic for agentic AI and vibe coding. In fact, the more applications that sit on top the worse the problem gets.

AI CEOs Get Cold Feet

Having greedily rushed to get their open generative AI services to market as 'minimum viable products' leaving all the shortcomings as 'externalities', the AI bosses spent this summer pretending to care as a way of demonstrating the 'true power' of what they'd foolishly unleashed. 

Altman signposted the fraudster's charter, and later found himself on the receiving end of a tragic wrongful death suit in connection with the death of a teenage user of Chat GPT between September 2024 and January 2025. This appears to have led the CEO of Microsoft AI to begin his own hand-wringing over the illusion that you seem to be having a 'conversation' with an AI, which he couldn't resist 'branding' as 'Seemingly Conscious AI'.

Humans replaced... then rehired

Meanwhile, overly enthusiastic adopters of chatbot functionality found themselves rowing back on their plans to nuke their customer service teams. Klarna performed such a volte 'farce', as did Australia's Commonwealth Bank, making it all the more bizarre that Microsoft should publish some, er, artificial research claiming to be able to 'predict' which jobs will be replaced.

Even if it were possible to run 'agentic AI' processes that do a lot of the mundane work "...before escalating..." more complex issues, to whom would they escalate those issues? Experienced senior managers? When they retire, who will have gained the experience to replace them? 

Advisory AI?

While the state of Illinois became the first to ban the use of AI to provide therapy, the UK government remain undeterred, announcing its decision to enable the unwitting British 'populace' to use agentic AI for everything from employment advice to obtaining driving licences...

Meanwhile, lawyers have had to be warned again about the fact that open generative AI tools produce fake law.

And before you start thinking of simple processes that AIs could fulfill, it's worth pointing out that ChatGPT-5 still fails at such seemingly straightforward tasks as creating an alphabet chart with each letter represented by an animal whose name starts with that letter; maps and decision trees, among other things. But a generative AI will still boast that it - or others - can do such things. For instance, when searching for an example of poor map making, Google 'AI Overview' produced the following slop (my emphasis):

The assertion that "AI can't do a map of Europe" is false, but it highlights the limitations of generative AI in producing accurate, detailed maps, which often contain errors like misplaced cities, incorrect country borders, and inaccurate iconography. While AI has access to a vast amount of data and can create maps that look plausible, it struggles with the precision and reliability required for a complex geographical representation.

AI Insecurity

But by far the worst issues are to do with security, including 'poisoned calendar invites' containing malicious prompt injections. This is a grave issue that 'better prompting' cannot fix, and the open architecture of open generative AI militates against a 'zero trust' approach which is unlikely to be commercially viable in any event. 

Rogue AIs can only be shut down

An 'agentic AI world' would be wide open to malicious prompt injections, hallucination, bias and error. So what, you might say. We could just shut it down. Yet researchers have found that some AIs can resist shutdown and find ways to keep working on their latest tasks. 

And if you've replaced your 'traditional' staff, systems and business processes with an AI, what then? 

Further reading:

For the AI sceptic's view, I follow Professor Barry O'Sullivan, Denis O., Axel C., Simon Au-Young and Georg Zoeller. Your mileage may differ ;-)

There's Nothing Intelligent About The Government's Approach To AI Either

Surprise! The UK government's under-funded, shambolic approach to public services also extends to the public sector's use of artificial intelligence. Ministers are no doubt piling the pressure on officials with demands for 'announcements' and other soundbites. But amid concerns that even major online platforms are failing to adequately mitigate the risks - not to mention this government's record for explosively bad news - you'd have thought they'd tread more carefully.

Despite 60 of the 87 public bodies either using or planning to use AI, the National Audit Office reports a lack of governance, accountability, funding, implementation plans and performance measures. 

There are also "difficulties attracting and retaining staff with AI skills, and lack of clarity around legal liability... concerns about risks of unreliable or inaccurate outputs from AI, for example due to bias and discrimination, and risks to privacy, data protection, [and] cyber security." 

The full report is here.

Amid concerns that the major online platforms are also failing to adequately mitigate the risks of generative AI (among other things), you'd have thought that government would be more concerned to approach the use of AI technologies responsibly.

But, no...

For what it's worth, here's my post on AI risk management (recently re-published by SCL).

Of Living The iLife, Dinosaurs and Data Portability

I'm not here to sound the death knell for Apple, but the announcement of the iCloud is a defining moment in the company's development. Will it remain a facilitator, or become an institution that exists only to ensure its own survival?

The 'cloud' or utility model for computing is not new. In fact, consumers have arguably held their data and basic applications 'in the cloud' ever since adopting public email services, blogging services and so on. What's new about the iCloud is the automated way in which all a consumer's content may be synchornised and otherwise 'managed' across all the consumer's (Apple) devices.

Seen from a hi-tech standpoint, Apple's move is typically bold and innovative. Yet the centralised omnipotence this may hand to Apple seems an attempt to reverse a 20 year trend toward enabling consumers to control their own data. In this sense, the iCloud appears to be the sort of product a major bank or telco 'dinosaur' would introduce in a last ditch effort to survive by locking-in its customers - and just imagine the complaints there'd be, given the switching challenges for consumers in those markets. So data portability is absolutely critical (along with personal data protection and security), if the iCloud is to be seen as a consumer 'enabler' rather than a predatory move by an aging institution.

But does the mainstream consider data portability to be important? I mean, I'd like to think that Apple's early, tech-savvy customer base would realise it's a bad idea to hold all your applications and data with a single provider, just as financially savvy folk realise the benefit of a fully-diversified investment portfolio. I have an iPhone and an iTunes account; but I also have a Toshiba laptop and a Dell PC. Those computers run Microsoft's Windows and Office package, and I have a Hotmail address; but I very deliberately browse with Firefox, blog via Blogger (Google), Tweet, hang out on Facebook and follow various blogs using Netvibes. And I use Spotify, not iTunes, as my main music service. In other words, I'm not going to let any one provider see, process, hold or control all my data - or even have a complete back-up or copy. That would feel closed and controlling, rather than enabling.

But, ironically, I suspect many people in the mainstream will see the need for software service diversity as a hassle or a problem to be solved by a single service provider, which is why Apple may quite genuinely see a market for the iCloud.

Does that make Apple a genuine facilitator or a dinosaur that's spotted a meal?

Image from eBandit.

Two Stones, One Bird?

Convenience.

Because most web sites with anything remotely important on them seem to require log-in codes, I keep many different usernames and passwords in my head. Apparently, the average person uses 12 (Independent Extra, 21.11.07, p.8). That's nothing compared to the many phone numbers that we used to remember before we began relying on the directory in our mobile phones and laptops, or Skype. But it hardly aids freedom of movement around the web.

To ease my passage, so to speak, the (worryingly named) Open ID programme would have me replace all my passcodes with a single ID. It would sit in a database somewhere to be checked when I access each participating web site.

Cue another standards battle, and Round 10 between Google, Microsoft et al.

But the people working on the semantic web would say that I shouldn't have to move around the web at all. Their goal is making information "understandable by computers, so that they can perform more of the tedious work involved in finding, sharing and combining information on the web". As I recall the explanation of Dr Nick Gibbins (School of Electronics and Computer Science at the University of Southampton) at the SCL's Law 2.0 event in September, the key issues are trust and provenance in the information which the computers are being made to understand. Both vary according to the source, time and context in which the information is given, as well as the content itself. You trust Prof Lillian Edwards' view of privacy law, but not her tips on car repair. But rather than drawing on a single ID in a single (hackable) repository somewhere, the computers would rely on a whole range of circumstantial evidence to confirm that the data in question is likely to be true and relevant to you - or in a log-in scenario, that the person whose computer is trying to gain access to a database is you.

Cue another massive battle over standards, but also over ontologies, taxonomies and other elements of the semantic web that are worthy of such top-draw words.

I guess that Open ID may be a stepping stone along the way to the semantic web, in which case we should get on with it. But that does seem like two stones for the one bird. Whereas the semantic web promises convenience without humans having to do all the moving around - so two birds with one stone.

I know which sounds better.