Why AI users must understand AI mechanisms

It’s Saturday afternoon, the sun is up, but the weather is so cold; fall is coming. I just got back from a day out and I am preparing a cup of tea, while my head is lingering on a topic that’s been in my head for a while. It comes as no surprise : AI.

This time, I share my take from an engineering perspective after a one-month experiment on coding AI agents. This article starts with a simple explanation of what AI is, then what distinguishes AI-powered software from classic software, and ends with the key takeaways.

I’m Rahenda, a product manager with a background in engineering, management, and business. I share my learnings on tech-related topics in Layman’s terms, turning complex topics into something easily understandable by a 12-year-old child or a 90-year-old grandma.

AI only understands 0 and 1 (literally !)

First thing first, let’s settle that computers work by processing 0 and 1 (numbers). Imagine it like a switch on (1) and switch off button (0). By stacking series of 0 and 1, computers can generate complex values (such as 3, 4, 9999, etc.), images, videos, words, and everything we’ve known so far.

Let’s call this language of 0 and 1 as “computer language”.

These are examples of how some English words are represented in computer language.

On top of that, we’ve built generative AI. Generative AI is built on mathematical probability. Here’s a quick example to illustrate it simply: If someone says “1, 2, and …” What would you say? Most probably, you’d say “3”. That is probability applied to AI. Based on a given input, AI guesses the next item that would most logically go with the input.

I use the term “logically” instead of “probabilistically” just to make the message easier to understand for ordinary people. The key message is: AI has been trained with historical contents of human interactions, which makes AI’s probability seem intuitively logical in the eyes of humans. Hence the “logically”.

But AI works like cake baking

Generative AI has a dictionary that allows it to understand any given input (such as a question) and guess the most logical output (an answer to the question).

Put simply, AI’s dictionary is like a series of stacked filters. Each filter narrows down every possible answer to the point that there is only one possible answer that passes all of the filters.

Think about the time you bake a cake. You use a sieve to filter the flour, letting only the finest part pass through. AI works the same way, as shown in the illustration below. It filters values to generate what is most probably the right answer.

One prompt, different outcomes

When filtering elements, we can’t control which ones go through and which ones get filtered out. Hence, AI’s answers are not always the same. To the question “What is the weather today?” The first time, AI might answer “The weather is sunny”. The second time, it might answer “The weather is cold but the sun is bright”.

This is what we call “probabilistic”. It means, AI is probabilistic because it guesses, not thinks like human.

You don’t always get the same answer

Software is omnipresent in our lives. Connected devices, smartphones, laptops, cars. Everything runs on software.

Before Generative AI, software only did what it was asked to do. Click on “connect to my smartphone”, and your car connects to your smartphone in a single click. Click on “update the inventory of factory A”, and the system resynchronizes with its database to fetch the latest stock state.

With Generative AI, “connect to my smartphone” might lead to the removal of your smartphone from the car’s list of trusted devices. “Update the inventory” might lead to data deletion.

Software engineering used to be dictated by a sequence of instructions. At the end of the sequence, the output is always predictable, as in the following example.

# Let's define 2 variables and their value
a = 2
b = 3
# The following will print "a + b = 5"
print ("a+b = ", a+b)
# Modify the value of one of the variables
a = 100
# The following will print "a + b = 103"
print ("a+b = ", a+b)

But with Generative AI, there is no guarantee of getting the same result across several executions. The same recipe does not always yield the same outcome, as the following example illustrates:

1. Generate a report on the latest inventory status
2. Update the inventory database of each factory
3. Generate a report on the latest inventory status
   
==> The 3rd instruction does not guarantee a report based on the updated information from the 2nd instruction.

Unlike classic software, AI-powered software is capable of much more complex tasks. AI can generate “creative” text, but it lacks the determinism that is key for critical systems, such as factories, or for “simple” tasks.

So, AI yes, but…

Does AI bring values to your business?

The answer depends on two questions:

1. What is the use case? → Think about the Job To Be Done !
2. What is the cost of a wrong answer? → In terms of finance and user perception

Some examples are explained below.

So next time AI pops out in your head as a solution, ask yourself:

• “what is the use case?”
• “What is the cost of a wrong answer?”