Enterprise data programmes have spent fifteen years getting good at rows and columns. Warehouses, lakes, pipelines and governance layers were all built around records that arrive with a schema attached. Meanwhile a growing share of what an organisation knows about itself now lives in messages, and almost none of that arrives with a schema at all.
The shape of the problem
A chat message is not a small document. It is a fragment of a thread, dependent on the messages around it for meaning, often edited after the fact, sometimes deleted, and frequently carrying an attachment that is the actual content. A single decision may span a voice note, three replies, a reaction that functions as approval, and a file dropped in without comment. Systems designed for transactions handle none of that gracefully.
Volume is the least interesting part. The hard parts are boundary and intent. Where does one conversation end. Which of the four people in a thread was accountable. Whether a thumbs up constituted sign off. Analytics tooling built for structured records has no vocabulary for any of it, and the instinctive response, which is to dump everything into a search index, produces recall without meaning.
Language is a further complication that structured pipelines never had to solve. The same decision may be recorded three times in three registers: a formal line in a ticket, a shorthand version in a channel, and a joke in a direct message that carries the real objection. Deduplication logic that works on invoice numbers has nothing to grip. Sentiment and tone, which are noise in a ledger, are frequently the payload here.
Why it stopped being a storage question
For a long time messaging was treated as ephemeral by default and nobody was seriously inconvenienced. That assumption held while email carried the record and chat carried the chatter. It stopped holding when the substance moved. Approvals, negotiations, escalations and instructions now travel through channels built for speed and never built for retrieval.
The consequence is that much of an organisation’s institutional memory sits outside every system with an owner, a retention schedule or a defined access review. This is not shadow IT in the old sense, because the tools are usually sanctioned. It is sanctioned tooling holding unsanctioned quantities of consequential material.
Regulators noticed before most data teams did. Duties framed around OSHA recordkeeping requirements in manufacturing and supply chain settings assume a written record exists and can be produced on request, which becomes an awkward assumption when the instruction that mattered was delivered as a voice note on a shift supervisor’s phone and never left the handset.
What the record has to survive
Vendors describe this as a modelling problem, and partly it is. Discussion of moving beyond models in enterprise machine learning has converged on a point that applies here directly: data quality now matters more than architecture, and the messiest data most organisations hold is the conversational kind.
The practical requirement is preservation with fidelity. A message stripped of its attachment is a reference to content rather than content. A thread stored without edit history misrepresents what was said at the moment it was said. A reaction discarded as decoration removes the only evidence that anybody agreed. Fidelity costs money, and it is the difference between an archive and a searchable pile that nobody can safely rely on.
Capture at the source, not in the memory of the user
Public sector records guidance reached this conclusion two decades earlier and states it plainly. Advice on how to capture it automatically notes that configuring messaging systems to capture messages and metadata removes reliance on individual users, and lists the recurring failure modes: messaging tools built without records functionality, limited search, difficulty tying messages to accounts or case files, and trouble capturing complete records with metadata and attachments intact.
It also warns that anything left unscheduled must be treated as permanent.
That last point reframes the cost model. Undecided retention is not neutral. It defaults to keeping everything forever, which is the most expensive option available and the one most organisations arrive at by accident rather than by choice. Indefinite retention also enlarges every future obligation at once. A larger corpus means a longer search, a wider breach surface, and more material to review the next time a dispute or an investigation requires somebody to produce everything relevant within a fixed number of days.
Retrieval is the real test
Storage only becomes valuable at the moment somebody has to find something. Guidance on how to search proportionately when responding to access requests is blunt that there is no technology exemption. An organisation must apply the same effort to retrieve archived or backed up material as it would when retrieving it for its own purposes.
The same guidance offers a worked example of two thousand emails in which the only personal information is a name and an address, a useful reminder that volume and relevance rarely correlate.
Meeting that standard requires three things unstructured stores usually lack. Adequate metadata. The ability to query across sources for everything relating to one person. And honest knowledge of whether material has genuinely been anonymised or merely had a column removed and a label changed.
Making the conversation the unit of record
The shift that resolves most of this is small and unpopular, because it means rebuilding capture rather than buying a better index. Stop treating a message as a row and start treating the conversation as the object, with a boundary, an owner, a retention decision and a resolvable identity for each participant.
Organisations that do it inherit a corpus they can ask questions of. Organisations that do not inherit a very large, very fast search box over material they cannot interpret, which feels like capability right up until the first time somebody asks a precise question and the answer has to be assembled by hand from four applications and a colleague’s recollection.
