AI is supposed to make everyone more efficient. Measuring that quickness is more challenging.
Mike Krieger knows new-stage tech well. He and Kevin Systrom founded Instagram — a tool that is now widespread in its business use — and joined Anthropic as chief product officer in 2024. Nascent tech often causes a bit of corporate FOMO.
On the “Superhuman AI: Decoding the Future” podcast, Krieger said that companies are starting to measure the productivity of these tools, rather than blindly investing.
“The best products can be grounded in some kind of success metric or evaluation,” he said. “I’ve seen that a lot in talking to companies that are deploying AI.”
Krieger listed the questions that companies should ask before investing in a new AI tool: “Is this a good product now, and is this a product that’s going to set up to succeed and scale?”
Putting a number to it is the difficult part, Krieger said. For some roles, like technical support and legal, he said that the metric was obvious: How much shorter was turnaround time on certain tasks? For other uses, less so.
“When it gets fuzzy, it’s very hard to then evaluate, did it help?” Krieger said.
Related stories
Business Insider tells the innovative stories you want to know
Business Insider tells the innovative stories you want to know
Google CEO Sundar Pichai said in June that AI had created a 10% boost in engineering velocity. A Google spokesperson later told Business Insider that he arrived at that figure by measuring the increase in engineering capacity created, in hours per week, from the use of AI-powered tools.
Two years ago, Krieger said that many AI tools didn’t have metrics attached to them at all. “They were driven by this AI FOMO that was happening in the the CIO suite,” he said.
That’s changed, though. Krieger said much of the enterprises he sees now are looking for return on investment, or at least an internal adoption metric to prove that it’s working.
“I often get the question, how do I know if Claude Code is working for my organization?” he said. “I often ask people just to look at the daily active metrics because those don’t lie.”
Usage metrics are booming, as bosses look more closesly at which workers are using AI tools and which are falling behind. Meta recently rolled out a dashboard and a game to track AI usage.
Krieger said that usage metrics may be a company’s best bet. “People do not use tools over and over again every day if they’re not providing value,” he said.
