Tag: AI limitations
-
When AI Research Agents Hit Reality: My Microsoft Researcher Experience
⋅
The promise sounded almost too good to be true: an AI agent that could seamlessly blend internal organizational data with web search capabilities to produce comprehensive, structured reports. Microsoft’s Researcher agent marketed itself as the solution to information silos, promising to transform how we gather and synthesize organizational intelligence. Naturally, I had to put it to the…
-
LLMs: They Know More Than They Let On (And That’s a Problem)
⋅
In a fascinating new study titled “Inside-Out: Hidden Factual Knowledge in LLMs,” researchers have uncovered compelling evidence of a significant gap between what LLMs know internally and what they can express in their outputs. This phenomenon, termed “hidden knowledge,” has important implications for evaluating and improving AI systems. The Knowledge Paradox Consider this scenario: You…
-
Beyond Words: Assessing LLMs’ Ability to Process and Reason with Tabular Data in PDFs
⋅
Large language models (LLMs) excel at handling text, but they can stumble when it comes to answering questions about numerical data in tables. To explore this limitation, I tested how advanced models handle two common scenarios: Scenario 1 – Multi-page Table in a PDF The task was to find employees earning between $70,000 and $100,000 (name and…