One of the things that keeps me up at night is how we are going to centralize all workflows through an AI hub. I want every single piece of information that comes in or out flowing through this AI-nexus.
But AI is inconsistent. Temperamental. Sometimes brilliant, sometimes dumb. I’m reluctant to entrust it as our central nervous system. Let me use some imagery to highlight the challenges.
Grok is generally my go to for images because it has less restrictions. I tried to use it to help with some images for this week’s Pensford newsletter. It failed miserably.
ChatGPT did much better. What’s really interesting to me about ChatGPT was that I drifted away from it for about a year because Claude had gotten so much better. But lately, the tables have turned again.
Speaking of Claude…
This didn’t surprise me because Anthropic (owner of Claude) has made it clear they are focused on business use cases and coding, not consumer/retail facing interactions. This works out well for me because we use Claude for all of our internal stuff. What I am not fine with is the recent throttling Claude has imposed regardless of which paid tier you are on - it just keeps telling me I am over the context window and halts all my work. That really pisses me off.
The one that continues to disappoint me the most is Gemini. Google had a giant headstart on ChatGPT and Claude. It has more data to train on than even Meta. Yet it keeps finding a way to suck. Also, please stop with the incessant prompts in Google sheets to analyze the data for me…
I’m using images to illustrate the point, but what if instead these were a spreadsheet?
Honestly, the one that scares me the most is hallucinations. AI is like me predicting interest rates - it just knows it is right, even when it isn’t.
Remember the best image from above? Here it is again - ChatGPT.
How many of you could pick Christopher Waller out of a lineup? Here’s his LinkedIn photo.
How good did ChatGPT do, really? Sure, they are both old white guys with grey hair. Obviously, nothing happens to my business because the image is slightly off. But what if it was an Excel output I was relying on to make a decision? It was close enough to make me think it was right. As any 23 year old might say, “that’s terrifying.”
To make matters worse, a few minutes later I asked ChatGPT to do it again and this was the result. WTF. Where’d the gold chain come from?!
We love Excel because the outputs are consistent and predictable. We trust Excel. That’s particularly ironic given the subject matter of the photos.
Internally, our best use cases for AI have been very tactical (abstracting loan docs) rather than aspirational/strategic. Our loan abstract has undergone a lot of prompt refinement, edge cases, nuance education, etc. It can’t generate an image or write a poem, but it can abstract a loan. I wondered what else in the realm of CRE it could tackle.
We ran a bunch of term sheets through our loan abstract tool to see how it would do. Remember, this is not a web interface - we own this model. I figured the results would be pretty wonky, but it actually crushed it. This was a first pass - not a single prompt refinement.
Sponsor - XYZ Borrowing Corp
Deal Name
We still QA’d the results (trust but verify), but it was spot on. I guess once you have loans down, term sheets feel like a warm up.
Until we trust AI for accuracy and consistency, we are laser focused on using it for boring old tactical stuff. The visionary stuff is gonna have to wait.
In the meantime, The Real Boss™ uses it to dress me for our upcoming Hornets season tix. Note the sweet slip on Skechers…