The Partner Test

Years ago, I had to attend a lot of monthly enterprise reporting meetings at a large academic health system. Each meeting had different groups attempting to unify several aspects of their reporting into one enterprise package. Each group had a well thought out methodology and had confidence in their data, and in the context of each group, they were correct. The issue was that at the most basic level, the definitions of each group were different, so there was no enterprise number that could be trusted. It appeared to be a reporting issue. It was actually a governance issue.

This is the lens I have when I evaluate an AI partner. We never addressed shared definitions and the clear ownership of the data behind the tools. The tools are temporary. The frameworks that shape how people understand and possess that data are more permanent, which is why the partner that a system chooses to work with is more important than the model they select to implement.

The Six-Month Promise

I was asked to evaluate a population health platform. This was early in my career, and my team was already deep in a troubled implementation. Our issue was the typical one, data from multiple disparate sources that needed extensive integration and curation before any platform could promise to offer us anything useful. The vendor reps selling us the platform claimed that a full data warehouse would be implemented in all the sources within 6 months.

I asked them where they built this before. Nowhere, it turns out. I asked them about the electronic health records they had engaged with. Their response was zero. It became evident after several similar kinds of responses that they were explaining a build they had never done. They did not know how our organization actually functioned, and thought they were more capable than a team of professionals who had built this several times before.

The writing part is what decided it for me. My leadership required me to fully assess the situation, meaning I needed to request a formal proposal from the vendor. I told them directly to avoid six months. They repeated six months in the proposal. It was also overpriced, assuming that my team would provide more personnel on top of what they were offering, to deliver something that was not possible to actually do in six months, or eighteen months for that matter. At this point the timeline was irrelevant, and I began to consider the judgment behind the proposal.

For my leadership assessment, I measured their six month promise against my experience. Years ago, we combined three data warehouses across three organizations into a single data warehouse. Even from that starting point, it took us eighteen months before stakeholders began to trust the data. Given our lack of maturity, and considering that the vendor had no relevant experience with us, the offer was declined. The contract was not signed. My team continued developing our data warehouse and over time it supported the analytics our population health programs required.

What the Pitch Does Not Address

Almost every consulting firm has an offering related to artificial intelligence, and most of their decks have similar language. When three proposals promise the same thing using the same language, the differentiating factor for a buyer will be the reputation of the firm.

In my experience, the model itself does not fail. The failure is typically related to who is responsible for ensuring the model is completed and who answers for it after the consulting engagement is completed.

The Hard Part to Pretend

The most concrete example developed during COVID19. Health systems needed the capability to see real-time movement of patients and the availability of beds. Something as simple as an available bed represented different things across different health systems. One location meant a physical bed. Another meant a staffed bed. Before any analytics could help, we needed to agree on the definition of the words and who owned the right to say them. Once that was accomplished, the organization understood the real capacity it had during a surge of admissions, when the leadership needed to make the decision of where to send emergency cases and how to adjust staffing levels.

Because of that, I pay attention to the first few questions a partner asks. The partners worth hiring are the ones who ask about definitions and ownership. The partners who ask about technology first are the ones who open with a model and assume everything that is essential to it is known. They skip the stuff that determines whether the work stands or not.

The same goes for Independence. An advisor who is indifferent to whether you buy a tool or walk away will tell you the truth about it. That is the kind of independence you want, but it is extremely rare because most people are in the business to make a sale.

The Tool is Rarely the Difference

I was involved in one of my earliest system builds when I developed a basic database tool for a back-office function at a hospital. It was pretty rudimentary, but it helped a group that was manually tracking discrepancies using workarounds on sticky notes. The system was successful because it nicely matched the way the users performed their job, allowing it to identify issues and save the users time.

The success of that system had very little to do with the software itself. It was developed with a strong sense of the users and their jobs, which, 25 years later, is still my focus for a lot of my work. A good firm will dedicate time and resources learning the job rather than explaining the new technology they are going to implement. If a firm really understands the job and how the data is created, the software and the technology that is going to be used is irrelevant.

What Actually Tells You

The scorecards can be ignored. The most trusted indicator for me is that a partner can identify a health system that went live and stayed live, and then hands you the phone. Lots of projects go live. A lot less are still used and trusted a year later, and that is what I focus on the most.

Fit to size is the other quiet test. A program built for a large multi-hospital system could inundate a small hospital. Likewise, if a partner brings the same operating model to every situation, they have not examined any situation closely. It seldom wins the pitch, but it often determines whether the work lasts beyond the contract.

Where I Land

In retrospect, technology had nothing to do with these decisions. It was a matter of judgment. I needed to ascertain whether the stakeholders comprehended the work and the data and whether they would be forthcoming regarding the necessary commitments. Therefore, I founded Hutchins Data Strategy Consultants with two questions in mind. First, do we have the data to comprehend the work? Second, is your team prepared to manage what we have constructed after our departure? Those are the same questions I would pose to any partner, including ourselves.

Context and Sources

This edition is drawn from my own experience across health systems. Related editions: The Chief AI Officer Mandate and The Procurement Trap.

Christopher Hutchins Founder and CEO, Hutchins Data Strategy Consultants