Synthetic data covers the cases a designer expects. Real operating histories hold the exceptions nobody planned for. The real test is which one your agent actually needs.
The case that breaks an AI agent is usually the one no prompt designer thought to write.
Ask a model to invent a difficult customer service case and you will have one in seconds. It will be coherent. The facts will show up in a sensible order. The customer will describe the problem more clearly than most real customers ever have.
Real work is rarely that considerate. A customer replies to a six-month-old thread about a different order. A partial refund was issued in the billing system but never noted in the ticket. The return policy changed between the first complaint and the final decision, and the agent has to work out which version applies before doing anything. Nobody designing a synthetic dataset would think to write that case. In a busy support team, it happens every week.
Why the real-versus-synthetic question is back
AI developers have already used much of the openly available text on the web. The next gains depend on data that shows how work actually gets done, and most of that lives inside businesses: in support desks, CRMs, ticket queues and document tools. That is why the industry is looking for new sources of real-world data, and many developers prefer data licensed with the owner’s agreement.[3]
Synthetic data is the obvious alternative, and it is genuinely useful. So the practical question for anyone building agents is not which side wins. It is which source covers the cases your system will actually face.
What synthetic data does well, and where it quietly fails
Generated examples let a team create controlled variations, stress a known policy boundary or stand up a test environment before any real records are available. The designer knows the expected answer, which makes grading easy.
That control is also the weakness. A generator reproduces its designer’s assumptions with impressive consistency. If the designer forgot that customers sometimes paste the wrong order number, so will ten thousand synthetic conversations. Generated users tend to be unusually articulate, tool responses unusually clean and policies unusually unambiguous. Each simplification makes the task a little easier than production.
There is a deeper concern as well. A 2024 Nature paper showed that models trained recursively on their own generated output lose information about the original distribution, with the rare cases in the tails disappearing first.[1] That is a specific result under specific conditions, not proof that every synthetic example is harmful. It is still a good reason to keep real, human-generated data in the mix and to track where training data came from.
Test the cases that are easy to miss. Concept illustration.
Real records bring their own distortions
Real operating histories are not a perfect account either. Employees take notes selectively. Systems lose context in migrations. A ticket marked resolved may hide a customer who simply gave up. And the recorded human action is not automatically the right one, so a mistaken exception should never become a target answer just because an employee entered it.
Coverage skews too. Support archives overrepresent customers who complained and incidents someone bothered to document, while silent failures barely appear. Real data needs its own provenance notes, coverage analysis and outcome review. Calling it authentic does not exempt it from evaluation.
Grade the end state, not the conversation
For agent work, a convincing final message is not success. Success is the right change in the right system under the right policy. Tau-bench, a public benchmark for tool-using agents, makes this concrete: it simulates users talking to an agent that has tools and domain policies, then checks the resulting database state against the expected outcome.[2] It also measures consistency across repeated trials, which exposes agents that succeed once and fail the next time.
That design is a useful template for comparing data sources. Define a set of tasks, measure the end state, hold the model, prompt and evaluation fixed, and change only the data. If you add historical examples while also switching models, you cannot say what helped. Inspect failures, not just the aggregate score. A source can add useful variety, add almost nothing or introduce a new weakness, and each of those is a real finding.
Getting licensed business records into the mix
For teams that want real workflow data, the hard part is permission. A company may hold years of support conversations, business workflows and engineering ticket history, but customer contracts, third-party content and personal information all shape what it can offer.
SourceX licenses business data directly from companies for AI training and evaluation, and the company signs with SourceX, not with an AI lab.[3] The supplier decides which systems, records and date ranges are in scope, a rights review checks that it owns the records and can license them, and a diligence packet documents source, rights, preparation and permitted use.[4] For practitioners, that provenance trail is half the value.
De-identification is the other half, and it has to preserve the workflow. SourceX de-identifies data through approved processing routes before it is used or shared, and its trust center says plainly that no method is perfect and that the agreement records that re-identification may still be possible.[4] The practitioner’s test is whether the case still makes sense once names, accounts and identifiers are gone. A refund dispute with the order history scrubbed beyond recognition has lost exactly what made it hard.
Mix sources, then prove it
Most teams will land on a blend: designed cases for known boundaries, licensed historical cases for the exceptions nobody anticipated, and evaluations that show which one moved the metric. The right ratio is an empirical question, and the experiment should be repeatable by someone outside the team.
If you run a company with years of real operating records, the first step is small. A fit check is a few quick questions, with no uploads and no commitment, and knowledge-work companies with 50 or more full-time employees at peak are the strongest fit. Start with SourceX. The messy cases in your archive may be exactly the ones nobody else can invent.
About the founder
Noah Loul is a four-time founder and the founder and CEO of SourceX, which licenses business data directly from companies for AI training and evaluation. He previously founded B2B Rocket, acquired by Blackpearl Group. His work focuses on building permission-based data infrastructure for AI.
Sources
1. AI models collapse when trained on recursively generated data (Nature, 2024)
2. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv 2406.12045)
3. SourceX: License your business data for AI
4. SourceX trust center
