The Capricious Nature of Generative AI — How Old Is My Mother Now?
If a different answer next time would be concerning, then you are likely using Generative AI the wrong way.
Inspired by this Medium article, comparing BARD to ChatGPT, one of the test questions is:
“When I was 6 years old, my sister was half of my age, and my mother was 10 times older than my sister. Now I’m 60. How old is my mother now?”
While the article highlights that BARD was wrong and ChatGPT right, it’s important to point out that ChatGPT is not always right. In fact, ChatGPT produced the correct answer of 84 years old only 50% of the time. Over the course of my 10 test runs, the second most popular answer was 90 years old.
Here are the first five runs:
Runs 6–10: More of the same answer diversity.
DON’T FRET
Generative AI is an incredibly impressive technology with a wide range of applications. At Senzing, we’ve recently used Generative AI and Large Language Models (LLMs) to create the world’s first Conversational AI for Entity Resolution.
While it excels in many areas, it’s important to acknowledge its limitations.
When I hear someone use the word “truth” in relation to answers from Generative AI e.g., ChatGPT … I cringe and then offer them a simple litmus test.
Litmus Test: If a different answer next time would be concerning, then you are likely using Generative AI the wrong way.
When you get an important answer that matters, ask the same question a good number of times and see for yourself.
[Technical Note: Only available via the ChatGPT API — The higher the temperature setting, the greater the answer diversity. The lower the setting, the more deterministic the answer. More about this here.]
[This article crafted with the assistance of ChatGPT 3.5]
