Don’t [always] believe the hype
AI chatbots are becoming increasingly popular and sophisticated, completely transforming the way we interact with technology and access information online. I use them daily. I teach them, and I build with them every week of my life.
A recent article from The Verge 🔗 compares three of the most prominent AI chatbots: ChatGPT, Copilot, and Gemini. Each of these chatbots has its own unique features and capabilities that set them apart from one another and this article has one of the BEST timelines I’ve seen yet of the progression & competition. Seriously, check it out.
I won’t repeat all the work The Verge put into that link but they highlight some key things, like capabilities, perception, integration and evolution.
As AI technology continues to advance at a ridiculous pace, chatbots are becoming increasingly integrated into our daily workflows. They can assist with a wide range of tasks, from writing and coding to research and information gathering and I’m 100% sold we should use them in more ways than we think.
You know this.
However, it’s important to approach the integration of AI chatbots with a critical eye. While they can be incredibly useful tools, they are not without error.
We have to consider their limitations and to use them as a supplement to, rather than a replacement for, human expertise and judgment.
Enter The Drama of Reflection
The AI community recently witnessed a dramatic turn of events surrounding the “Reflection” language model, created by Matt Shumer this last week.
My Twitter feed was going wild.
Initially, Reflection generated significant buzz due to its claimed ability to outperform leading models like ChatGPT and Claude in various benchmarks. Shumer made bold assertions about the model’s capabilities and origins, stating that it was based on the latest Llama 3.1 model from Meta and utilized advanced techniques like reinforcement learning, problem solving and self critique.
The internet went wild of yet again, the next big thing.
However, as the community started to look more into Reflection, inconsistencies began to surface:
– Reflection was found to be based on an older less performant model
– Shumer’s claims about the model’s training and capabilities were questioned and not quite able to be reproduced
– Benchmarks and comparisons to other models were scrutinized and information was just plain ‘strange’.
As the truth about Reflection came to light, many in the AI community felt misled and let down. Even VentureBeat updated their article 🔗 welcoming you to meet “most powerful open source AI model in the world”.
Trust in Shumer and the Reflection model quickly diminished, and the hype surrounding the model deflated. Some even accused Shumer of intentionally deceiving the community for attention.
Why am I telling you this? Because as fast as this industry grows we simply can’t believe all the hype all the time.
I’ve heard this quite a few times:
- “Jim have you tried this 2-hour-old thing I found”?
- “thoughts on x vs y vs …”
- “why not post more breaking news?”
The Reflection drama is a reminder that we need to prioritize transparency and verifiability in what we take seriously.
This is why I send this newsletter every week.
You won’t find me as a contributing author at The Verge anytime soon, but I take a big responsibility in sharing my educated opinions and findings to save you time, money and hopefully make your day a little clearer.
I felt it on my heart to be transparent this week about how intense and quick to evolve this industry is every week.
My hope for you this week is to take this short story, think about ways you’re using or considering using AI chat technology – and be sure you’re well informed, and have someone in your corner keeping you on track that you can trust.
Tweets That Matter
You have to check this Twitter thread of an AI app that understood the importance of timing.
Trends these days are what bring eyes, interest and paying users. This is an absolute masterclass and seeing an opportunity, making a hard process easier than others, fast, and being rewarded for it.
