Hi, folks.
Welcome to another Engineering Evolve.
My name is Tom, the host.
I know it's been a while since we've done the
last one, so apologies for that.
We are back bigger and better than before.
Today, we're going to talk
about the fact that your
users really don't care if it's AI or not.
They're more interested in getting stuff done.
So without further ado,
as we dig into this concept
of the concern about
AI or whether something's
vibe coded or is it nicely
curated by a human,
we're going to talk about
various different points.
But AI,
machine learning, all that type
of stuff, of course, has
been around for many decades.
If we dig into some of the more obvious stuff,
the fact that you have a junk filter and the
fact that you're various bits
of mail, you never get
to see because the junk filter
exists is exactly that.
This is how machine learning
aids us on a daily
basis.
Now, things like spam filtering,
fraud scoring, map rerouting,
all that type of stuff, it
is all machine learning.
It is all AI.
If you want to use a broader term for it,
I've been around for quite a while now.
Back in the day, we called
it machine learning, which
lived in the data science and
data modelling ecosystem.
These days,
AI is the term and that largely
also encompasses LLMs.
But I bet without putting
much thought into it, you
could name a whole bunch of
different places where ML,
maybe not obviously, but gets
used under the hood to
offer up a more cohesive experience
to the user, learning
from data that's gone before
so that you can have
a better experience today.
We're going to dig into this.
Welcome to Engineering Evolved.
Let's get going.
When we're talking about AI
in the current ecosphere,
what we're talking about for
all intents and purposes these
days is powered in some way or driven by or
curated by an LLM.
AI powered means not much
more than that to the
average user.
Now, of course, in reality,
there's lots of different
bits of data science and a
lot of it hasn't changed much in the past 20
years, like an awful lot of modelling,
linear regression stuff,
all those types of things.
They've been around since the dawn of time.
They're mathematical models
at the end of the day.
But of course, these days, with
the additional compute power
that all the larger companies,
all the medium to large
size companies have access to,
everything can be AI powered
and AI driven because compute
is cheap, relatively speaking.
But it does mean that when
you see everything is
being powered by AI,
it's a bit of a misnomer in terms of how
people are trying to sell their software.
Everything these days becomes AI
powered, AI driven because teams
are selling the architecture
instead of the outcome.
So if you're trying to
get onto the hype cycle,
then naming your thing AI
powered will make a large
difference.
Whereas leveraging AI to drive
a specific part of your
platform or even to drive
the development of your platform
is vastly different,
you know, as a consumer
to the person doing the
building.
And I think a lot of companies
are losing sight
of the fact that, to a degree, no one really
cares if your thing is AI powered or not, as
long as they get the answer they want out of
the far end.
And if they don't get the
answer that they want,
it doesn't matter if it's
AI powered or not, they
will go a different direction.
And that's, you know, that's
really the crux of everything
it's like on the market today.
Like companies feel concerned
that they are not jumping on
the hype train, they want
to get their AI driven,
AI powered stuff out there.
And I'm not for a second
saying that they shouldn't
leverage, you know, some elements
of LLMs or machine learning,
you know, statistical modeling processes
to get additional functionality and
support into their platform.
But not everything needs to have a big fat AI
powered label stuck on it.
And that I think is my take at the moment.
Let's hear from our first sponsor.
Okay, back at it.
So, of course, from an
engineering perspective,
where do LLMs come in?
Useful and how can you leverage them properly?
Now,
LLMs, of course, are guessing machines.
They guess the next word in a phrase of stuff,
assuming that you're doing language stuff.
It is a large language model.
So they're going to guess
the next word and they're
going to pick out common
phrases and come up with
something that feels like it's right.
And of course, for things like
repetitive things like coding,
they are excellent at doing so.
Now,
the real reason that we're
talking about this is that
your users grade you on
whether it works the same
way twice and not on the impressiveness
of the model.
So you might have a super expensive
Opus model running
somewhere.
But if you can't get the same answer out of
it twice for the same question
you're sticking into it,
then users are going to lose
trust in your platform.
Now, of course, there are many
different models and different
models cost a different amount
of money depending on what
you're trying to achieve.
And so finding that area where you can tune it
up and pick out a model that works well for
the exact question you're going
to ask it is important
from a software development
and engineering perspective.
If you just chuck every question at an Opus or
a Sonnet or a chat GPT for or, you know,
there are so many different
models that depending on what
you're trying to do, you
really need to figure out
the correct model for that question.
And there's a tradeoff between cost, speed,
inference, you know, and when
you stick them all together,
what gives you that ability to be able to be
confident in the answer that
the LLM is giving you
so that you can then hand that back to your
user and have your user
trust the output of the product
that you're selling them
because they don't care whether
or not it's an LLM.
They care whether the output
that you are feeding them
is the correct one.
So like, you know, as you're doing
your software development,
evaluations, fallback paths, latency budgets,
and then like things like graceful
degradation when chat GPT
or anthropic for the 600th time
this month decide they're
just going to drop offline,
what happens to the product
at that point?
And then, you know, also
from that what happens as
we have seen last week, for example,
if a frontier model that
your pipeline is built on
decides to get rug pulled,
and then suddenly you don't
have access to it.
All of this needs to be built in.
And so whether you're using
a frontier model or not,
you have to be able to deal with the ability
for it to not be there because of why your
customers are not going to be
impressed if suddenly you're
not returning responses because
the API is down, or if
you're hosting it locally
and it's fallen over.
But to that point,
the requirement for not running
or not the requirement, but
the desire to run a frontier
model might land you
in greater jeopardy than picking
a model off the shelf,
tuning it up, tweaking it, and
then hosting it internally.
If it is called criticality,
sorry, if it is critical
to the operation of your system,
if it if you can't deal without it and then
you're depending on a third
-party service for an API
for another, then good luck
because you're going to have
to deal with outages and your
customer is getting rather
pissed off.
So, you know, this is the type of stuff where
you have to be able to factor all this type
of thing in when you're engineering
your solution.
And that is, you know,
day one stuff.
Like,
if you cannot engineer yourself
around the idiosyncrasies,
the model upgrades, downgrades, rug pools,
different services, different providers,
you're going to have problems
down the line when you've
got customers on the system
and no ability to be
able to deal with outages
or that type of stuff.
So I'm going to leave that one there.
But you know what I mean, like, you've got to
think about all this stuff upfront,
because I don't know
if you've seen anthropics
uptime, but it isn't great.
And if they also like, you know, if Fable have
been around a couple of weeks
and people have built
stuff on top of it, and then it got pulled,
there'd be absolute uproar.
And that's all the type of stuff that you have
to be able to deal with.
Of course,
the cost of the label can also be an issue.
AI can be a trust liability.
And you know, people brace for
hallucination and for being
trapped away from a human.
And you can't charge an AI
tax because pricing follows
outcomes and not transformers.
So a few things to sort of unpack here.
AI, as we said at the start,
can definitely cause
a few interesting issues when
people know about the fact
that you're using AI inside of your platform.
Now, some people will not
care, most people will not
care.
But you also have to brace for the fact that
occasionally,
your application is going to hallucinate
or potentially return information
that is incorrect
and have it displayed to a human.
So what can you do about that?
What can you do to ensure that you give the
AI the most information for it to be able to
return a valid answer?
And what happens when it
seems to be erroneous?
So of course,
you know, you can put guardrails in to try and
track erroneous responses or things
that don't conform to the
output that you're expecting.
You can also
deal with hallucinations by feeding
it more accurate data.
Now, inside of organizations,
there's a mad rush to be
able to get everything AI -ready,
AI -centric, you know,
good to go for an AI perspective.
Now,
that is great.
But at the same time, if you are feeding it
junk data, then you're going
to get garbage in, garbage
out to a degree.
I mean, hopefully, your LLM
can work its way through
it.
But if you can guide it using APIs or data
entry that give it a better
chance of being able
to find the data and focus in on the data
that is important,
it will be cheaper from a token
spend perspective, and
it will be more accurate for your end users.
And so if you can deal with that, that will
obviously get you further
down the line to being able
to ship a product that has AI inside of it.
Now, when it comes to actually shipping,
people do not care whether
or not you've got AI
in it.
Just because you've got
an AI bot inside of your
application these days doesn't mean
that people are suddenly going
to pay twice the price
because you've now got a
whole bunch of tokens that you need to be able
to, like, you know, price for when it comes to
actually shipping your product.
So you can't charge an AI tax on top of
the application.
But what you can do is use LLMs to give
a more effective and efficient
outcome that will allow for
users to be happier and generally,
you know, more comfortable
using your platform.
And that, at the end of the day, is what
we're trying to drive with
AI usage, is to either
give them more insight or give them a way of
being able to interact with your platform
without having as much oversight
overhead and things they have
to do.
And so if you can turn that
LLM integration into
an outcome,
they are going to be more comfortable
paying for whatever
they have to pay for rather than you trying to
force an AI integration down
their throats that, you know,
looks like a tax because you're
suddenly going to have
to charge them more because
people are going to be
burning tokens.
So I hope that sort of makes sense.
Like, you know, you've got to be able to deal
with these types of things from within
a product ecosystem and figure
out how you're going to
integrate this stuff into a
platform and then also make
it available to the users
because all of that type
of stuff
is not just as simple as like, yes, I'm going
to integrate this platform, like,
integrate this LLM into my
platform right now.
You need to be able to figure out how you're
optimise it and also tune
it up for the operation
that you're trying to give it.
And that is not as simple as like, oh, I'm
just going to stitch Claude
into my workflow and then
magically it's going to start working.
So bear that in mind.
And now a second message from our sponsors.
Okay.
So now going back, just
take your step back and
thinking about this sort
of, you know, inherently.
There is, of course, a
line where users do care
that AI is being used in
a platform because data
provenance, data usage,
you know, is a human seeing
the data and pretending
that AI isn't in your platform
when it's making the
consequential decisions that a
user, a consumer of your product
is then going to act upon means that you have
to be
honest about the fact that
when you're using AI inside
of a platform, you have to tell them and you
have to be upfront about
where it's being used and
how it's being used to be able to offer them
the ability to challenge the
output that comes with your
application and pretending it's invisible
when it's making consequential decisions
is also a failure in itself, I believe,
you know, and so making sure that there is a
real, you know, a real definition
about and upfront, honestly
about this is where we use
AI in our application
to be able to offer you more
features, functionality and flexibility
in what we do is something
that you have to be able to do and have
to be able to tell, you know,
your customers about
because that type of stuff
is going to be important
to them down the line.
So hopefully this has been sort
of whirlwind episode, but
hopefully this has been, you know, a bit of a
thought provoking episode.
It's like, how do we leverage
LLMs inside of an
environment today?
And there are so many different
ways of doing it.
How do we add I like, but instead of asking
how do we add AI,
start asking what's now instant
that wasn't because of course,
the difference between
a ML model and an LLM is
worlds apart.
Like you don't have to do all the data prep,
you should do a lot of it, but you don't
have to do as much of the data prep as
you did before.
You can ask questions that before
would take absolutely ages
to get right and have the answers in front of
you, you know, within seconds.
And so how can we leverage
all the knowledge that
these LLMs contain and have
been taught and then make
the products that we are building
more efficient, more effective
and more available to our
customers and to our end
users so that they get bang
for their buck rather
than an organization just adding
a powered by AI tagline
into something because they
don't want to miss the train.
That is the question that
everybody should ask it, it
should be asking, not anything
else, but like, how do
we add value?
How do we make this available
to our customers in
a way that they would benefit?
Because everybody looks at chatbots,
but of course chatbots are
the possibly one of the
most inefficient ways of
interacting with an LLM.
If you have to sit there
and type everything to
a chatbot, you're going to
waste years of your life
just like asking a chatbot and correcting it.
And so from a product development
and product design perspective,
the question of course now
is how can you implement
an LLM and add value to your
product without having
to sit there and have a
user type everything in
and then read the response back.
And those who really excel
in the AI space will
figure out how to square
that circle and, you know,
get to a position where they
are capable of integrating
an LLM into a platform without
having a user having
to look at yet another chatbot.
Take that one away with
you, have a think about
it from a product development perspective,
an engineering perspective, how
would you integrate an LLM
into what you're trying to
do but without creating
a chatbot to have to deal
with it.
When you think, when you
figure that one out, go
build it.
I hope you found this useful.
Thank you very much for tuning
in and listening to
the latest engineering evolve.
My name is Tom, we will be back soon.
Thank you very much again.
Bye for now.
Thanks for listening to Engineering
Evolved with Tom Barber, where
ideas meet innovation and
leadership drives change.
If you enjoyed today's episode,
please leave a rating and
review wherever you listen.
It helps more leaders discover
the show and keeps the
evolution moving forward.