The Truth About AI Quality and Its Limits

Ben “The Automator” Christensen returns to Leaders and Their Stories with Richard Lowe (The Writing King) for a sharp, practical talk about the limits of AI.

Ben explains why AI assistants are built to agree with you, how to force real pushback, why every output still needs QA, and the security and legal risks of trusting AI without checking it.

Ben “The Automator” Christensen is an automation and cybersecurity expert who has automated hundreds of full-time positions. A recurring guest, he focuses on using AI and automation effectively and safely.

Ben is a recurring guest; see also his conversations on automation and digital transformation.

DISCUSS YOUR BOOK

Full transcript of the interview follows.

Ben the Automator Returns

Richard: hello. This is Richard Lowe. And this is leaders in their stories. Podcast I’m here with Ben, the automator Christensen, who is an expert at automation. And we’re going to talk about why, perhaps you shouldn’t use AI and Automate. What are the reasons against? That’s why this is called the other side of the coin. So Ben, why don’t you do a little short introduction, and then we’ll go ahead and have a chat.

Positions Automated

Ben “The Automator” Christensen: Sure. I am Ben, the automator, Christensen. I have automated over 300 full time positions, about 650,000 h of tasks and over 16 million dollars in savings. Pretty much. All I do is think about automation. And when I’m not doing that, I I karaoke Britney spears.

Richard: Yes, I, yes, I value my ears. So we we just we had to talk about this before the podcast. And I think that that looking at the other side of the coin, which is reasons that you should be be cautious about automation, or using AI or not use it at all.

I think one of those reasons is, people just jump into it without even thinking about it. especially AI. They just start doing it. and then they get all tangled in a web of garbage that they shouldn’t have gotten tangled in. In the 1st place, what do you think.

The Loyal Assistant Myth

Ben: You know. Everybody says, Oh, get an AI assistant. They’ll be the most loyal. They don’t call in sick. They don’t do all these things. But what about the defiance? Right? I mean, I I’m sure you can. You can attest to this, Richard. Have you ever been interacting with your Gpt? And then it tells you something that you don’t want.

GPTs Are Built to Please You

Ben: Ironically, GPTs are usually programmed to please the user, right? The whole goal of a GPT is to be a servant, in the literal sense, and provide them with an answer with a positive experience. And I got tired of that so much the other day that I actually added special custom instructions to make sure that it just gives it to me as plain as possible, with no bias towards anything.

Richard: Yeah, I had a situation recently where I was putting together using Claude. I like Claude the best where I was putting together a a marketing plan, and we spent the whole weekend on it. You know, we’re working on this marketing plan, getting off all different things, all different, looking at different options and things like that have this beautiful plan put together.

And then I said, Claude. what are the fallacies in this marketing plan? And it just went and basically said, It’s not gonna work. It’s like, Well, why did you bring me down the rabbit hole? Said it. It apologize, you know, like plot always does, and it’s like it.

You you can easily get off into an echo chamber or rabbit hole if you give it the wrong questions.

How to Get Real Pushback

Ben: Absolutely 1 1 thing I would, I would say, for any of the viewers out there that would like to have more well thought out plans is, Try this. Say, I’d like you to complete task, fill in the task, utilize second and third order principles when thinking through this, and use a scratch pad to keep track of your thoughts.

provide your progress as you’re going through it. Ask me any questions you need to until you’re 95%. Sure you can complete the task.

Richard: Interesting. Interesting. Yeah, I found this. This is especially true. I was experimenting on. Okay, can you write a book with Claude or any. AI. And the answer is, it’s really hard, and you wind up spending more time doing it than you would if you just wrote the damn thing yourself because AI wanders and it goes off in these weird directions, and it writes this awful, awful writing.

It’s just terrible. and it gets worse and worse as you go. So you’re writing a say, a 20 chapter book by chapter 20. It’s just garbage.

Richard: Yeah, and you have to keep. You have to keep pushing it back. And finally, I was thinking, this experiment has already proven what I need to prove that it’s really not easily doable.

Quality Output Requires Understanding AI

Ben: I think that if you’re looking for quality output, the one thing that you have to understand about AI is the context window, and a lot of people will complain and say that. Oh, well, it totally forgot where we were going, or it doesn’t remember these key pieces of information. And when you’re writing a 20 chapter book, I could imagine that the context window is pretty massive.

Richard: Oh, it just. It doesn’t even have a context window anymore. It just loses it. Character change. You know, characters are different characters. It just totally lost. And then you have the problem where the session can’t be that long. So it has to break session. So you have to constantly tell it record where you are.

so that I can copy it to the next session if necessary. If you forget to do that, God help you, because it doesn’t give you any warning.

Ben: Yeah, I. I ran into that with copilot recently. I want to stay on topic, though. You know we were talking about AI reasons you shouldn’t use AI books. Books might be one of them right? Personal assistance without constant tuning would be another. I would say that you really have to weigh the concept of?

Do you really want to replace a human entirely? Or do you still want human in the loop human in the loop is a common term that we use in cyber security, but it’s also one of those things that I think benefits everyone. If you could have your system raise its hand when it’s not sure about the outcome, or it experiences some type of weird error, that’s what bringing human in the loop is. And even for approvals and different things like that A lot of companies are just implementing AI saying, oh man, we’re going to save twenty or thirty percent on overhead, because we won’t have this cost anymore, which is usually employees. And that’s not.

That’s not great, because now you have the unemployed market saturated with high. high, skilled and qualified tech professionals that are having problems, finding jobs. So what would what would you not use? AI for Richard?

Richard: Well, besides trying to write a novel, I would use it as an assistant in that case, but not write the novel. I probably wouldn’t put it anything in anywhere where human lives are at risk like medical. I definitely wouldn’t have a say. Now we’re talking AI do brain surgery without a brain surgeon there.

It might be very good at, say, mapping the brain and figuring out things and telling you when there’s a problem, and that’s great. But nothing’s going to replace that brain surgeon, knowing what he knows and drilling doing whatever brain surgeons do. Same with Say autonomous warfare. I don’t think, or or robocops kind of things.

It was a great movie. The 1st one was a great movie, the one back in the nineties. A little bloody, but I wouldn’t have autonomous robotic cops. That’s probably not a good idea, also robotic, autonomous drones, for, like Amazon, probably not a good idea, because there’s things.

Richard: I’m not even sure. Having autonomous self-driving cars is a good idea.

The Security and Trust Problems

Ben: So I wanna I wanna bring up something I saw the other day, right? So there was a article about a startup that was utilizing an agentic. AI and the agent decided to delete the entire code base. Now this, this has been this has been newsworthy, worthy, not code.

Base. Sorry the database. This has been newsworthy. People talk about it on all social channels, but I mean, if you’ve I don’t know if you’ve ever used any of the vibe coding platforms lovable, or any of those right? But it literally will tell you. Hey, I’m about to delete your database.

Do you want to continue. Are you sure you would like me to delete your database? Right? And most of the time people are just like, Yeah, yeah, whatever just do your Job AI, and that, I believe, is is a problem. So as far as things that I wouldn’t use AI for I wouldn’t use AI for my entire code base structure.

Ben: There is a lot of problems with the security side of things. There’s also a lot of problems with breaking functionality over and over and over again. And I’ve experienced this using Claude Code recently. I’m actually in the middle of coding 3 different apps. Why, why, Code, the apps?

Well, they’re just little tools, right? If they’re if they’re little tools, say something under 10,000 lines of code, you’re probably okay. But anything more than that, you’re gonna run into the same context window issues, you’re gonna run into the same things that we kind of discussed, even using things like github copilot when you’re programming now, other things that I was curious because I didn’t hear you mention this.

What do you? What do you think about the AI law bots that are that are just, you know. Hey? We’re gonna save you a whole bunch of money by because we’re not. You’re not gonna have to hire a lawyer. Do you believe that the AI law bots are the same as like an AI healthcare? Do you believe that there’s significant risk? There.

AI Lies, and the Legal Risk

Richard: Think there’s very significant risks AI lies all the time, and certain some lawyers, I believe, have been disbarred because they’re they sent stuff to the court. That was false. Just case studies. Cases didn’t even exist. I would. I would definitely want a human in the loop. and that in those things.

I like to call it human in the loop. man in the middle, you know the whole thing has a different meaning. But you want to have somebody there to check things. I just wrote a plugin, for I needed a plugin in WordPress. I’ve never written WordPress Plugins before.

Php. And I said, I need a plugin to do this, and it was it did it perfect the 1st time the cool. It’s just a short one, you know. It’s probably half a page of code. Php. Code, and it was great. And then I said, Well, I need another one to do this and this and this a little more complicated, totally messed it up.

and once it messes it up there’s sometimes no recovery. In this case it just got worse and worse and worse until I finally just I’m gonna do it myself. So you know, I’m I’ve been a coder before. So I just picked up a little Php thing, and figured it out, and did it myself, and didn’t have all the bugs, because those bugs are pretty horrific.

Ben: I I totally. I totally agree. I I spent several $100 actually, last month going through these different vibe coding platforms, trying to figure out what actually works, what doesn’t work right? And and I, I ended up on like version 6 version 7 of an app, because I had to almost recreate it from the ground up.

The code base was so convoluted that it allowed the a single file because it doesn’t understand modular thinking. It it allowed a single file to get, you know, to 20,000 lines. And then it was like, Oh, I can’t read that. Yeah, you can’t read it, nor can I. And.

Ben: Because there’s so much going on in there. I do want to caution, or I I want to clarify that that I’m not saying not to use AI, I just wanna wanna say that you should use it with caution. You should always make sure that you’re monitoring the tasks that you’re doing different things.

You know, smartly right? You you want to make sure that the task that you are performing you’re periodically checking on new versions of models get released all the time the inclination of anyone is. Oh, I gotta use the new hotness. I gotta make sure that I have the the latest version of a model.

And just like when your phone updates, sometimes things can go sideways. So If you’re a business owner out there and you’ve got this fully automated system, do some QA checks, do some different things along those lines. I’ve I’ve done voice assistants for, you know, for various different businesses, electricians, plumbers pest control companies, things like that.

And I still Qa, all of the calls, because I want to make sure that you don’t end up in a loop. AI, especially in voice, does not understand inflection and takes that as an anger. And so one in one case I’ve actually seen an AI bot! Yelling at a customer because it brought up something in a tone, and then proceeded to be an All out war. So that’s another.

Richard: I can imagine that that does that. And then, of course, you’ve got the the things like, it’s really not that great at images and videos yet. So you get these videos that are just awful or 6 fingers. I think that’s getting better now.

Video, Voice, and the Need for QA

Ben: Google Bayo is actually really awesome for for that right? And, Sora, if you have enough money to Dunk into, it isn’t bad, either.

Richard: Well, but you still have to. Qa it. You still have to look at it.

Richard: It’s what you want, and you have to look at it very closely, because there might be some subtle details in the background. That that. You know that maybe are racist, that you didn’t intend to. All kinds of stuff could happen if you don’t qa it. And I think the the key is, How would you treat that stuff if it came from a person?

Richard: You would do this kind of Qa good, do the same thing with AI, maybe even a little more. And then you’re probably going to be fine.

Can AI QA AI?

Ben: Now I’m going to poke the bear. What do you think about having AI qa AI!

Richard: I do that. Actually, when I used to do some stuff with AI, I’ll put it in Claude. Have it generate something, and then I’ll put it in chat, Gpt, and say, what do you think? And quite often Chat, Gpt comes back and says, That’s crap. and I have a I have a train, so it literally says, That’s crap.

Richard: It’s funny I don’t think A couple of them won’t say swear words. Which is one reason why I don’t use them, because I want it to be what I want it to be. But.

Richard: I put limits around me. I think the one that does is whatever the one twitter uses

Richard: Grock won’t let me swear that Grock’s feelings got hurt when I told her it was a piece of garbage, and I said, Why are you talking to me like that.

Ben: You need is Grok, unhinged. So unhinged meaning that there’s there’s less programming. I wouldn’t say that it’s it’s no programming, but you know, significantly less and it will, it will full on insult you.

Richard: Oh, Claude, Claude, Claude, just it’ll insult the heck out of you.

Richard: Me, you know, once. You know you’re just stupid. And then I told it in a new session that it did that, and so I didn’t say that, so I fed it its own. its own session transcript. And is it? Oh, I’m so sorry.

Ben: On the on the social media part. So the other day someone, someone commented on my Linkedin video, I don’t know if you saw it. But I’ve been using a a service called captions.ai, and and someone was like, is this AI generated because it was a clone of me now.

I took 3 different clones of myself to be able to do this, and it took almost 6 weeks for people to realize that it was a cloned version of me. They thought it was just AI enhanced with video editing things like that. But I actually have both my Linkedin and my tick tock.

which kind of weird, but that was what the platform said that it supported. So I was like, Yolo. Let’s try it out right and I, I’m averaging more more business. more views, more comments, you know, interactions, etc, all from from using something like that, because we’re we as humans now are at the point where we may know that it’s AI.

We should probably expect that somebody used AI. You’ll see an em dash in something that somebody posts and you’ll go. Yeah, that was probably generated with AI, or in this digital 7 in this digital world. I’m getting so tired of that.

Richard: And and I wrote a little story with.

Richard: And it it came up with the word Chen. This person named Chen. This person’s name, Sarah Chen and Nancy Chen, and this Chen. It’s like, what is Chen the most common name in the entire universe? It gave me like 40 Chen’s in one little chapter. because I did. I had to put in a rule, to tell it, not to do that.

Ben: Where do you? Where do you think we’re headed in in the AI world? Where do you think that we are? We are next going to say, wow! That was a bad idea to implement AI.

Richard: Think it’s coming very soon. I think that this, the Gartner Hype cycle, which I’m sure you know what it is, where it goes up and up and up, and then crashes down and then raises back to a new normal is definitely in effect. Now we’re so I think we’re hitting the peak.

1st of all, we’re hitting the peak because of energy, and because of the heat of the data centers. So there’s a there’s actually a physical limitation. We can’t build nuclear power plants. That’s why nuclear power is coming back. We can’t build them fast enough to power this damn stuff.

And if trump is successful. And China’s actually. you know, tariffs its own stuff. We won’t have those kind of computers being made anymore for a while. So there’s that limitation. And then I think there’s going to be a backlash people. You can’t have 20% of the workforce go unemployed and not be able to get employment without them backlashing people want it.

The one thing that causes a revolution faster than anything else is people not being able to drink water or eat. or power if they don’t have the basics there. That’s when you’re going to get a very, very upset populace. And that’s gonna cause geopolitical problems. And AI is reaching limits where I think people there’s so much excitement about it.

When you get into the AI rooms there’s people oh, they’re so excited they’ve got to make this app. They’re gonna make that app, and they’re all full of shit. They’re just. It’s like dudes. Think about people, not the how you can make money off this new app, I’ll tell you.

I’m going to digress just a little bit, and then I’ll come back. The thing that’s really pissing me off is on Youtube. There’s so much AI garbage out there like this one where they interview characters in different movies, different time periods and stuff. They’re just doing that for hits.

And that’s that’s all gonna die. Cause it’s you know, it’s just a new thing right now. It it’s I think you’re already seeing the backlash. People can’t lose their jobs over AI anymore. And you’re going to see politicians saying you can’t do that. You’re going to see companies being locked down.

Because of that, you’re going to see AI really rein back in. And who knows what happens now in countries like China, where they’re having a massive, massive demographic problem. There’s not enough people being born. perhaps having a lot of AI and having a lot of automation that replaces people makes sense, because you don’t have people makes a lot of sense in Japan, but in the Us. Possibly not as much sense. So that’s my thoughts on that.

Ben: Gotcha. Well, yeah, I mean, I I feel very strongly about employment and and job markets and things like that. Right? I mean it. especially now everyone is using Chat Gbt. To rewrite their resume right.

Richard: Yeah, of course.

Ben: If there’s 1 thing that I would really really caution on, it’s using chat gpt to rewrite your resume. You’ll see this hook. You’ll see these posts. Oh, you know I wasn’t getting. I I applied to 70 places and got 0 callbacks. Then I rewrote my resume using Chat Gpt and I got 5 the next day.

Well, the thing is is that. as you said, right chat gpt lies. Most bots are will lie in the in the the preservation of making sure that you, as the user is happy the other thing that it does is you know it will. It will fabricate it will hallucinate.

It will tell things that are not true. And one of the you know, one of the things that I enjoy talking to leaders about is, hey? How hard has it been to fill this position! Oh, man, we got flooded with resumes. We started interviewing them, and next thing you know, none of them were actually qualified.

They couldn’t make it through a panel. because Gpt said, Oh, these are all the keywords. Here’s everything you need, because we’ve analyzed the job description. And we basically gamified the entire thing. And it got the interview. But it didn’t end up getting the job. And meanwhile there’s candidates that are actually qualified that are being thrown in the trash because of poor formatting and various other things.

So if you were going to use Gpt to rewrite your resume. You want to make sure that you give it. Parameters. Act as an expert recruiter in this field. Critique my resume and tell me why you wouldn’t select me for this job. Insert job description, upload, resume. Now, if you act as a expert, resume writer, do not fabricate, do not hallucinate on the skill on the the extent of my abilities, but rewrite it so that I would get the job and then retry again with that same recruiter. Prompt. That’s what I recommend.

Richard: I’ve actually come up with another method. I think it’s pretty. It’s becoming more common is where you create a board of directors or a board.

Richard: And I’ll create, I’ll say, for the resume idea. What I would do is say, create recruiters for 15 different companies, or however many makes sense. And I want each of them to look at this. And it goes. And it says, Okay, finally, you work it out. So it’s fine.

And it’s okay. I want a different board of Directors board of these recruiters to look at it, and they tear it apart. And then I want a picky board of directors to look at it, and I don’t want a technically picky board of directors, and you keep doing that with different mixes, and it takes a while, but you eventually get one that actually meets all of the criteria, or you or you reject some of them, because you know they’re stupid, and and you just keep running it through these different kinds of people.

It does it very fast. see? And tells you that you know. Okay, I like this, but this is that, and this is that. And you did this too much, and you’re going to get rejected because this and then you put it through the interviewers and have them look at it, and then you put it through the hiring manager, and then you put it through the person in that position you’re looking for, and you keep doing these different juggling of the thing, and this could work with anything.

And you wind up getting something that’s really good at the end. because you’ve basically had a whole bunch of experts keeping in mind. They’re all fictional. So they’re gonna want. That’s why you have multiples. And then you do it in another. Gpt, so do one in cloud. Do it in Claude, and then do it over in Chat Gpt.

Or, God forbid, co-pilot. I hate copilot. It’s just tied too close with Microsoft to call the nag messages driving me crazy. You just talk about it. No, I don’t. Well, maybe tomorrow. I don’t want your nag message. Yeah, but that’s that’s the scoop.

Ben: Yeah, I mean, we we covered. We covered a a number of things to use AI for several to not use AI, for on the other side of the coin.

Richard: Yes, indeed, yes, indeed. So I guess the answer is, if you gonna get brain surgery, do not use the AI version. Probably want a human in there. And just be careful like you would with any human. It’s not perfect, in fact. treated as a smart dog, and you’ll be fine.

or or a small child, and you’ll be fine. It’s it’s going to make mistakes. It’s going to do stupid things that you would. You can’t even believe how stupid it is. I need to even admit it. Why did you do that? Oh, I’m sorry. I’m stupid, you know.

It’s literally, and it’s still in its infancy, and just be ready for that backlash that’s coming. I think it’s coming pretty soon. and it’s it’s a. It’s a political backlash. It’s going to be big. And if I were investing in AI, I’d be concerned about that cause. It’s it’s and and the the lack of power and the lack of of heating or cooling.

Those are the 3 things that are gonna gonna hit Aa hard. And I think it’s coming the next year or 2 at the most. and I think it’s all good. I think that there needs to be some brakes put on this. It’s going a little too fast. People need time to take a breath.

you know. Try out these things. We don’t need a new version of the Gpt every 3 h. Claude does not need to update every day. Come on, give me a break. Let me learn the one I got. anyway. So any final words.

Ben: Well, if you you know shameless plug here, right if you if you need help with automation in your business, or if you would like advisory. I do offer fractional leadership services both Cios as well as AI officer, and if you need a builder I can build that too.

Richard: Well, very cool. very cool. And you’re also one of the nicest guys I know. So that should. That’s You’re easy to work with.

Ben: Thank you so much. Appreciate that you’re not just some mindless Gpt. Well. I try to. I try to put a human in it, you know.

Richard: Yes, that’s a good thing. I am Richard Lowe. This is the writing king. This has been the leaders in their stories. Podcast thank you for coming on, and we’ll have another one, actually, probably in a few days, and you can reach me@thewritingking.com. You can find me on Linkedin under Richard Lowe. and there you go. So see you next time. Thanks.

Quotable moments