At Business Travel Show Europe, Oversee’s Sam opened a panel with the question now sitting in front of every travel operations leader: how do you know whether AI is actually working?
He was joined by Ted Capeless, Oversee’s VP of Sales and Account Management; Torsten Kriedt, Principal of AI Value Creation at Provectus; and Danielle Cavnor, BlockSkye’s VP of Global Sales.
For years, measuring new technology was relatively simple. Run the process, confirm it worked, and move on. The tool either passed or failed. AI does not fit that test.
Can You Measure AI on a Pass/Fail Basis?
Danielle challenged that mindset directly. TMCs have to move beyond the pass/fail lens, she argued, and “shift from measuring the action to really measuring the outcome.” The metric that matters is not how many rebookings a machine handled. It is how many travelers actually took the trip the AI itinerary produced.
Ted picked up the thread. People often assume AI runs the entire process, he said, but “it’s a tool,” one of many inputs in a larger workflow. The AI may read a request, pull policy, and gather missing details before handing a prepared case to an agent. The agent’s judgment still determines the outcome. Grade any single step and you learn very little.
That is why the pass/fail verdict breaks down. Oversee reached a similar conclusion during its BTN Group webinar in March, where the discussion turned into a five-dimension AI scorecard: productivity, quality, containment, traveler experience, and resilience.
What Happens to SLAs When AI Answers Instantly?
Sam put the old scoreboard on the table. For years, TMCs measured service by speed: how fast agents answered calls, cleared queues, and turned around replies. When AI can respond in seconds, those metrics start to lose meaning. Compliance can look near-perfect while saying very little about whether the traveler actually got what they needed.
Torsten challenged the premise directly. Once speed is essentially free, he said, “the whole SLA discussion is nonsense” at the tactical level. The reason is simple: “no traveler cares about how quickly the phone call has been answered if the answer is not the right answer.” Ted agreed: “It doesn’t matter how fast you answer the question, if the answer is wrong.”
That is where the measurement model has to change. Speed still matters, but it is no longer enough. TMCs need to measure the quality of the answer, the accuracy of the recommendation, and whether the traveler’s issue was resolved correctly. In the scorecard, that sits under quality.
What Should Agents Do With the Time AI Hands Back?
Ted started with the tasks no one should need an experienced travel agent to handle manually: a traveler asking for an itinerary, a frequent flyer number, or a basic booking detail. These are routine lookups. They require no judgment, create little value, and generate no revenue. AI can take them on and “complement the process and make it very efficient,” freeing the agent to focus on work that requires a person.
Torsten pushed the point further. When an agent’s only job is to look up an answer the system already holds, the agent is not adding value. That work should move to the machine. Instead, he wants TMCs to “upskill your travel agents to become more of a personal assistant in the real human world,” focused on the moments travelers cannot solve alone.
This is the productivity dimension of AI, but not in the narrow sense of headcount. Productivity means capacity. The same team can handle more work because the tedious, repetitive volume is cleared off the top.
Danielle gave the idea a name: super agents. AI that supports the floor during live calls, paired with twenty years of booking experience, “is going to just perform overall better for the company.” The technology does not replace the person. It makes the person harder to beat.
What Does “Getting It Right” Look Like for Travelers?
Danielle moved the lens to the traveler. Success comes down to one thing: “whether or not they get it right.” Everyone has dealt with a bot that asks for information it should already know. Travelers judge that interaction by the answer, not by the speed of the response. That is the traveler experience dimension.
A weak setup returns five options, four of them irrelevant. A stronger setup returns one recommendation that fits the constraints that actually matter: the right travel window, the right airline or alliance, the right policy, and the right context.
That is what the containment dimension measures. Not whether AI touched the task, but whether it resolved the task without creating more work for the traveler or the service team. None of that removes the need for a person. A traveler in crisis still needs a human, Danielle said. But AI paired with an experienced agent can serve the client better than either one alone.
Does the Operation Hold Up When Everything Goes Wrong?
Resilience separates AI running as a feature from AI running as infrastructure. Disruption is where that difference shows. Danielle gave an example from BlockSkye’s own operation. An in-house tool monitors thousands of RSS feeds by the minute, looking for early signs of disruption. When conflict broke out, the system flagged the risk in time for the team to staff up before traveler demand surged. The AI did not resolve the crisis. It bought the team time.
That time matters. When AI absorbs routine volume during a spike, agents can stay focused on stranded travelers who need empathy, judgment, and fast decisions. An operation that only performs well during calm weeks has bolted AI on top. One that holds up during disruption has built it into the operating model.
The panel also moved into an area the March session had not covered: governance across markets.
Language is increasingly solvable, Torsten noted. Data residency is not. An EU traveler’s data cannot simply reach a US-based agent unless the right processing agreements are in place. At BlockSkye, Danielle said each customer’s own DPAs define the boundaries. For global TMCs, resilience is not just about disruption response. It is also about whether the AI operating model can hold across regions, customers, and regulatory environments.
How Should a Travel Manager Measure the TMC?
Torsten reframed the whole question at the end. A travel manager should not focus on grading the AI tools themselves, he argued. The TMC is an extension of the travel manager’s role. The real question is whether the TMC helps move the program forward while reducing the burden on the internal team. His test was simple: “How can you help me solve X?” If the TMC clears the pain point, whatever runs underneath is the provider’s responsibility.
Ted named the failure mode. Too many providers treat AI as “checking a box that says I’ve got AI” and then market it, instead of asking where AI actually earns its place. When AI gets a travel manager to the outcome they came for, “it’s a win, it’s a success.”
That reframing sets the ceiling for the whole panel. Productivity, quality, containment, traveler experience, and resilience measure how well the operation performs. Whether the TMC solves the buyer’s real problem explains why any of it matters.
The Scorecard TMCs Actually Need
The pass/fail verdict made sense when technology either fired or it did not. AI requires a more complete measurement model. The panel’s message was clear: measure the outcome the traveler received, not just the volume an agent cleared or the seconds a reply took.
The five-dimension scorecard from Oversee’s March webinar puts structure around that shift. Productivity, quality, containment, traveler experience, and resilience give TMC operations teams a way to prove whether AI gains hold up quarter after quarter.
Oversee built AgentSee to be the layer that earns those numbers. It can cut average handling time by 50% or m ore, take on more than 70% of ticket handling, and lift productivity by up to 90% in selected processes.
To see how it fits your GDS and service workflows, book a walkthrough.