The new data science question: What should AI be allowed to change?

When every dataset can become a hundred analyses, control matters as much as creativity.

One dataset, many possible analyses

Let’s say you have some data.

For example, it could be a spreadsheet from your company, a database full of user events, a bunch of medical records, or a TCGA gene expression dataset to make it more specific to life sciences. You want to understand what is inside it, and what insights you can generate from it.

A few years ago, this usually meant choosing one analysis pipeline and then spending a lot of time making it work. You connected the data, cleaned it, transformed it, ran some models, made some plots, and then wrote down what you found. An overwhelming amount of time was actually spent on the first part, that is dubbed data massaging for a reason.

But with AI, the situation becomes a bit stranger, since most of the technical parts are done by it. This opens up the possibility to scale whole thing horizontally.

You no longer have to run only one analysis.
You can run hundreds.

And these analyses do not have to differ only at the end, where the AI writes a different summary. They can differ from the very beginning. One analysis can use different data connectors. Another can clean and join the data in a different way, thus approach the problem from a different angle. Another can use another statistical method. Another can generate a completely different type of insight.

So the question is not simply:

“Can AI analyze this dataset?”

The better question is:

“Which parts of the analysis should be fixed and later make it deterministic and well tested, and which parts should be allowed to change dynamically by the AI to explore the possibilities and find novelty?”

That is the main idea behind I’d like to communicate here to you.

The analysis pipeline recap

For the sake of illustration, imagine that every data analysis has a few layers.

First, there are the connectors. These decide where the data comes from. Maybe we use one database, or five APIs, or uploaded files, or an internal company system.

Then comes the data massaging. This is the ugly but important part. We clean the data, remove bad rows, normalize values, merge tables, filter things, calculate new variables, and make the dataset usable.

After that comes the actual method. This could be a statistical test, a machine learning model, clustering, survival analysis, dimensionality reduction, pathway enrichment, or whatever else fits the question.

Finally, there is insight generation. This is where the results become something a person can use. A report, a recommendation, a ranking, a warning, a hypothesis, a figure, or a list of next steps.

The interesting part is that each layer can be fixed or flexible. We don’t want something that works to be flexible enough to risk that the LLM will hallucinate something.

You can fix the connectors but allow the methods to vary.

You can fix the data cleaning but allow the AI to generate many different interpretations.

You can fix the final report format but allow the system to search through different data sources.

Or you can let the whole thing vary from beginning to end.

That last version is the most open-ended. It is also the hardest to control.

A simple example if you are looking for an apartment

Imagine you want to choose a new apartment. You have some basic data: rent, size, location, commute time, noise level, safety, sunlight, and energy costs.

A normal analysis might rank the apartments with a fixed scoring system. Rent gets some weight. Commute time gets some weight. Size gets some weight. The apartment with the highest score wins.

This is already useful but it is only one possible analysis.

An AI system could do something more flexible. It could connect to rental websites, public transport data, map data, neighborhood statistics, noise maps, and maybe even weather or sunlight information. It could also process the same data in different ways.

One version calculates rent per square meter. Another estimates the real commute during rush hour. Another removes all apartments where the total monthly cost goes above your limit. Another tries to estimate which apartment you will still like after six months.

Then the method can change too.

One analysis finds the cheapest acceptable apartment. Another finds the best balance between price and comfort. Another groups the apartments into types, such as cheap, convenient, quiet, or risky. Another tries to find the apartment that is least likely to become annoying later.

The output can also change.

One report might say:

“Choose Apartment A. It is the cheapest option that still meets your basic requirements.”

Another might say:

“Choose Apartment B. It is not the cheapest, but it saves you the most time every week.”

The format of the output can also vary, one might prefer a pdf report, another person prefer in a HTML format.

All of these can come from the same starting data.

So what should we do?

Maybe we decide that the data sources should be fixed. We trust only a few apartment websites, but we allow the AI to try many scoring systems.

Or maybe we do the opposite. We allow the AI to collect data from many places, but we force it to use our fixed personal scoring model.

Or maybe we fix the final output. Every time, the system must give us three answers: cheapest, most balanced, and best long-term option.

This is already very different from simply asking an AI to “analyze the data.”

We are designing which parts of the analysis are allowed to move, and which ones to hardcode and make reliable.

An example from cancer research

Suppose we want to analyze TCGA gene expression data. We want to find genes, pathways, or patterns that are related to survival, cancer subtype, or disease biology. This sounds like one task, but it is not.

At the connector level, we have many choices. We can use only RNA-seq expression data. Or we can add clinical survival data. Or mutation data or any other modalities. The input can even be derived raw results from another machine learning algorithm.

Already, the analysis can branch in many directions, depending on what hypothesis we have.

Then comes preprocessing.

Do we use raw counts, normalized counts, TPM, or FPKM? Do we log-transform the values? Do we remove low-expression genes? Do we correct for batch effects? Do we adjust for tumor purity? Do we split the cohort by subtype? What do we do with missing clinical variables? Each of these choices changes the result.

Then comes the method part.

We could run differential expression analysis. We could run univariate Cox regression. We could run multivariate Cox regression adjusted for age, stage, and tumor purity. We could make Kaplan-Meier groups. We could cluster samples. We could run pathway enrichment. We could build a classifier. We could look for gene modules instead of single genes.

And after all of this, the AI still has to decide what kind of insight to produce.

Maybe it writes a biomarker discovery report, or pathway-level interpretation or maybe it produces a clinical risk hypothesis. All of these analyses can begin with the same TCGA cohort. But they are not the same analysis.

What should be fixed and where should we allow the AI to be creative?

This is where the design choice becomes important.

If we allow the AI to change everything, then it can explore a huge space of possible analyses. This can be useful for discovery. Maybe it finds a signal that we would not have checked manually.

But the result can become messy if every analysis uses different connectors, different preprocessing, different methods, and different insight formats, then comparing the outputs becomes difficult.

So we can freeze part of the pipeline.

For example, we can fix preprocessing:

“Always use the same RNA-seq normalization, filtering, and clinical covariate handling. After that, let the AI explore different biological interpretations.”

This gives us stable data handling but flexible insight generation. The important part is that this “fixing” can mean that we actually write a deterministic code to do that part of the analysis, or we use LLM agents, but strictly define the constraints in what it can do.

We can also fix the final insight format:

“Always produce a survival-biomarker report. But allow the AI to test expression, mutation, methylation, immune infiltration, and external validation data.”

This keeps the report comparable while allowing the data sources to expand.

Or we can fix the method:

“Always use multivariate Cox regression adjusted for age, stage, and tumor purity. But allow different feature-selection and preprocessing strategies.”

This makes the statistical comparison cleaner.

Why this matters

Most people talk about AI data analysis as if the main benefit is speed, but that is only part of it.
The more interesting benefit is that AI lets us explore many possible analytical paths at once, and let’s us test many many different hypotheses.
But this creates a new problem. If everything can vary, then we need to decide what should not vary, since we actually want to understand what we are doing.

In some cases, we want discovery. Then we should allow more flexibility.

In other cases, we want reproducibility. Then we should fix more of the pipeline.

Sometimes we want to test robustness. Then we keep one part fixed and vary another part on purpose.

The important part is that:

You can outsource the technical implementation to the AI, but you cannot outsource your understanding.

And to help with understanding, you need to know that what part of the data analysis you can trust, and which part can introduce LLM errors.