Share

For more: jasonsteiner.xyz

TLDR and Introduction:

This article is a follow-up to Virtual Cells1 which aims to provide a bit more technical nuance to the concept of simulating biology, or at a minimum, of creating virtual cells.  The goal is to give some intuitive frameworks in a semi-technical manner for thinking about neural networks, programming with data, and limits of complex computation, particularly as they relate to biological systems.  The intuitive gap between biology and computer science remains significant in my experience, so this is an effort to create more bridges.

The topics that will be covered in this article are the following. Feel free to jump to the sections via the links below:

1. Programming with Data

2. Multimodality

3. Limits of Computability of Sequences

4. What about PINNs? (Physics Informed Neural Networks)

5. A Moment of Transparency

By the end of the article, I hope that the reader will have a greater intuition for the ideas behind deep learning particularly as they relate to complex systems like cell biology.  The ideas presented here are not exhaustive nor decisive, but perhaps they will be useful.  

Let's jump in.

Programming with Data

The AI wave has brought with it the idea of programming 2.0 -- that is programming with data instead of explicit instructions.   In AI, this has been characterized by the move from "expert systems" -- where researchers tried to encode decision trees and explicit rules that were derived from human experts -- to deep neural networks which learn by mathematically optimizing some prediction error between an objective and a calculation.  The latter is accomplished through the application of large amounts of training data to arrays of tunable parameters and the result is a program -- a set of instructions that can be executed by a computer.  In this sense it is the same as the programs from expert systems, except instead of human interpretable if-then-else statements, these decisions are embodied in an inscrutable set of math equations. 

The most intuitive way of understanding this is the following:  one of the core tenets of computing is the conditional statement -- the if-then-else statement. There are many other concepts used in programming, but this is among the most fundamental and is at the root of the CPU.  In all historical programming, these statements have been defined by the programmer to be explicit.  This logical foundation is one of the reasons that computing is such a powerful and scalable architecture, particularly for anything digital where binary true/false statements are easily encoded, executed, and copied.  However, the challenge of specifying such rules for systems that have not been constructed from the same logical foundations, for example, many natural and biological systems, has been considerable.  These are systems in which we know neither the scope of variables nor the extent of their interaction dynamics necessary to craft such programs. 

To be sure, the field of synthetic biology has tried with some limited success.  The majority of such attempts have been aimed at creating programs with differential equations to the extent that a cell can be considered to be executing a program (one ultimately defined by its genetic content and directed by its environmental context).  However, these approaches are inherently limited by a whole host of unknowns such as rate constants, concentrations, locality uncertainty, and others.   It is a tall order to make this explicit, even to the extent that dimensionality reduction can simplify the matter.

What Does it Mean to Program with Data?

Deep Neural Networks as programs is a pretty fascinating idea.  A neural network that possesses many layers is essentially a very long math equation of multiplications and additions.  If this were all the neural network consisted of, it would only produce a good linear classifier.  The underlying reason is that regardless of the number of layers if all calculations consisted only of multiplications and additions, the whole network could be reduced to a single linear layer that simplified the result.  The introduction of a non-linear function into the network changes this fundamentally.  The non-linear function effectively creates an "if-then-else" statement at each neuron.  If the previous layer's inputs, times the weights, is above 0, for example for the common ReLU function, then pass the signal on, if it is below 0, do not pass the signal.  Effectively, a binary true/false statement.

Figure with References2:  A deep neural network has many layers of interconnected neurons that are calculated via multiplication and addition — this is the part inside the () in the equation. If f() were linear, any arbitrary number of layers could be compressed into a single linear layer and the ability of the network to calculate the output of an associated input would be limited to a linear classifier. With f() as a non-linear function, for example, the ReLU function shown, each neuron effectively becomes an if/then operator which sits at the core of conditional logic and computing. Every layer thus acts as a form of nested conditional statements vastly increasing the “expressivity” of the neural network to compute complex logic.

This switch provides the fundamental unit of programming and one can then easily imagine a deep neural network as a very large array of if-then statements -- which can produce extremely complicated logic. This logic is not, however, produced by any specific external intelligence.  The parameters are defined solely by the mathematics and goals of the network and influenced by the environment the network is exposed to -- in particular the specific training data.  There is an interesting parallel to biological systems, particularly for pre-trained neural networks.  Both have an underlying circuitry that reacts and evolves in response to external stimuli.   One actually might muse that a high-dimensional version of the Waddington landscape is not dissimilar from a training loss landscape, albeit current neural network architectures are likely much less plastic, one might imagine that combinations of generation, search, recombination, and reinforcement would yield much more dynamic evolutionary neural networks. 

Figure with References3. The plot on the left represents a training loss landscape of a neural network as you move through parameter space — some sets of weights produce a much more accurate calculation of the objective function (blue states). The goal of training a network is to find a path to the lowest point which minimizes the training loss. A similar view can be seen in stem cell differentiation where the parameters of the neural network are represented by the epigenetic status of the cell.

So, if we consider training a neural network as programming it with data -- in the end we end up with a program.  The goal is that this program is a faithful model of an underlying distribution, but this largely depends on the data. 

Multimodality

Programming neural networks with data can be useful for many tasks in cell biology, but it's easiest to think about them as either encoding or decoding tasks.

• Encoding tasks take data and aim to extract features of that data for classification and prediction tasks.  These tasks generally have a preliminary endpoint in "embedding space" just upstream of the desired task.

• Decoding tasks generally take a point in embedding space and aim to generate or reconstruct a real word sample.  

• Combined Encoder/Decoders are used for end-to-end predictive or translation tasks such as how a drug might affect a gene expression profile.  The initial inputs need to be embedded and then decoded. These can be auto-encoders which are generally used for compression and removal of noisy signals, or they can be translators between modalities.  

Using multiple different types of data in a neural network is based on the principle that different types of data carry different pieces of information about a system. For LLMs, it's simply that we interact with text, audio, and vision seamlessly so the modes should be interchangeable or at least integratable.  For cells, it's that a single modality doesn't fully represent the reality of a cell so we should collect more.  There are a few ways of thinking about the role of the role of multimodality in biology -- one is that a multimodal data set will be more representative of a cell state, another is that we may only be able to practically measure one, so we want a predictor for the others.  In both cases, the underlying data must have some underlying relationship.

Encoding as Compression

The most intuitive way I have found to understand multimodality is for its utility in compression.  When a single data type is being used to program a neural network, the resulting embedding can be considered a compressed version of that data -- in essence, it is a smaller version that has summarized all of the relevant information of the underlying data.  It naturally will depend on how far you want to summarize which is a parameter of the embedding size and will determine the utility of the summary. Consider summarizing a book — if you wanted it to be useful, you’d need some minimum text length, but you would not need the entire book. Now consider the case where you had two different types of data, for example, a book and its corresponding film. Both of these are digital files. If you concatenate them, once they are converted into a tokenized form, they appear effectively the same to a neural network (independent of the original data types). During compression, one would expect that information in one data set would be used to compress information in the other, and vice versa.  For example, if you were using a simple byte pair encoder, the frequency of bigrams in one data set would influence the compression in the other. The same applies if you were using more complex compression such as convolutional filters or self attention. If the two files are related to each other in reality, this yields a richer and better compression of the full state.  

Figure: For a very nice overview of compression in unsupervised learning, Ilya Sutskever provides an accessible talk on this4

If the two data files are not related, you may still end up with a reduced compression, but it does not reflect a state in reality -- the resulting embedding is not useful.  A precarious feature of neural networks is that they can fail silently -- meaning you may see a training convergence that may be meaningless.  Efforts to avoid this issue have been developed that include conditional covariates, but the basic principle is the same.

As it relates to encoding tasks for cells, the more data that can be concatenated in the input, the richer the embedding representation can be — for example representing multi-omic cell states. In practice, this has had limited success largely because paired data is not particularly abundant. Some efforts to proxy this have been to use the central dogma to “translate” one mode into another, for example, gene expression to proteins5, but this is a much less accurate reflection of the true matched proteome of any given transcriptome. One area that has made some progress in high through paired data generation is in spatial transcriptomics, which can pair images with gene expression.

Encoding as Prediction

A second way to look at multimodal data is as a predictive task -- for example, can you predict one item from another?  Typically, these methods operate in the representation space -- where the goal is to predict the representation of one modality by the representation of the other.  The most straightforward method of doing this is a method called CLIP training (Contrastive Learning Image Pretraining).  This effectively takes two different modalities and trains two embedding models for them to minimize the difference between paired embeddings and maximize the difference between unrelated embeddings.  When fully trained, the paired modalities will have a single embedding that both inputs share -- for example, a gene and its corresponding protein.  In this way, if you have a new scenario (which is under the same circumstances as the training scenario) and you can only measure one modality, you can predict the other.  The difficulty lies in ensuring that the inference scenario is the same as the training scenario.

Figure with Reference6. The basics of Contrastive Learning is to create a common embedding that represents two related data types — for example, text and images. The goal is to optimize the alignment (the blue diagonal) of the two embedding models for matched pairs and minimize them for unmatched pairs. The result is a single embedding that can be used to represent either data type. Whereas text and images are static distributions, in biological systems, trained embedding models are only valid within their training domain so to make valid multimodal predictions, the inference condition has to be similar to the training condition.

A more comprehensive method of multimodal data has been proposed by LeCun's Joint Embedding Predictive Architecture (JEPA)7 model which can be viewed as an extension of CLIP to include learning a function in the latent space that translates one embedding to another.  One of the benefits of this approach is that it can smooth out noisy experimental data through the embedding process.  JEPA is an Energy-Based Model (EBM) which is a model that assigns an "energy" to the relationship between two embedding variables depending on the degree to which they are related to each other — essentially the predictive loss.  This is used in the context of video, for example, where two frames of a video need to have a continuous representation but there is an infinite number of possible video continuations, so the "energy" assigned to each frame relates to how well it continues from the previous frame.  At inference, you then simply follow a low-energy trough through representation space to generate the scene.  This approach requires progression or time-resolved data.

Figure with Reference (see 7). The JEPA model is a more general form of learning in embedding space. The figure description has the details.

Developing simulation models in embedding space is particularly useful for systems with noisy experimental data because it can reduce system noise, however, it depends on the ability to effectively create embeddings. In the context of cell biology, embedding pairs could be unimodal pairs — for example, two different -omics, or complete cell embeddings, for example representing a cell’s differentiation or disease progression in a frame-by-frame manner.

Limits on Computability of Sequences

In the context of virtual cells, it’s important to clarify what the goals are. The perspective of this series is that the goal of a virtual cell is to be able to simulate trajectories in time — and this requires some degree of time resolution which is an area of considerable data paucity at the moment for both technical and practical reasons. Simpler versions of a virtual cell are more akin to endpoint prediction or classification tasks — for example, predicting whether a drug will kill a cell or not. Such models may be useful for endpoint modeling but have much less resolution on the actual biology.

The view of neural networks as programs brings with it the limits of theoretical computation and specifically as it relates to simulations, limits on the computability of sequences.   There are several concepts related to the idea of sequence prediction that relate to compression and information theory.   For a detailed review of this, I would refer the reader to Machine Super Intelligence, specifically Chapter 5 on the Limits of Computational Agents8.

One of the key principles is the idea of the minimum length of a program that can produce a specified sequence -- this is known as the Kolmogorov complexity.   For example, in the case of physics, let's say we want to produce the sequence that represents the distance of a traveling object from a point.  The "program" that does this is simply d = v * t.  In general, physics equations can be viewed as low Kolmogorov complexity algorithms -- they are designed to model and faithfully produce a series of outputs of arbitrary length.   

A more complex example would be an algorithm that can produce the digits of pi.  For example, an algorithm known as the Bailey-Borwein-Plouffe Formula can be used to rapidly produce the n-th digit of pi.

Figure: The BBP algorithm is a compressed way of calculating any arbitrary digit of pi without the need to calculate all prior digits.

If we look at a deep neural network that is designed, for example, to produce a sequence of words at a given perplexity, the “algorithm” used to produce those words would be the entire network.  To the extent that a smaller network or a shorter algorithm could be used to produce a model with the same perplexity, that would tend toward the Kolmogorov complexity of the language or the shortest program that can be used to produce a sequence.

Without going into the full detail of computability and prediction, below is a snapshot of the conclusion from the above reference:

We have shown that there does not exist an elegant constructive theory of prediction for computable sequences, even if we assume unbounded computational resources, unbounded data, and learning time, and place moderate bounds on the Kolmogorov complexity of the sequences to be predicted. Very powerful computable predictors are therefore necessarily complex. We have further shown that the source of this problem is the existence of computable sequences which are extremely expensive to compute. While we have proven that very powerful prediction algorithms that can learn to predict these sequences exist, we have also proven that, unfortunately, mathematical analysis cannot be used to discover these algorithms due to Gödel incompleteness.

These results can be extended to more general settings, specifically to those problems that are equivalent to, or depend on, sequence prediction. Consider, for example, a reinforcement learning agent interacting with an environment, as described in Chapters 2 and 3. In each interaction cycle the agent must choose its actions to maximize the future rewards that it receives from the environment. Of course, the agent cannot know for certain if some action will lead to rewards in the future. Whether explicitly or implicitly, it must somehow predict these.

So why does this matter for the future of virtual cells?  Firstly, this is a mathematically derived statement about the ability to generate predictors for complex sequences.  To the extent that we want to model cell states over time, we may consider that to be a complex sequence.  This argument states that any such predictor will necessarily be very complex -- i.e., likely a very large neural network.   If indeed this is the case, then such a network will likely require a commensurately large amount of data to train, less any inductive priors that can be added to constrain the network.

The second point that is made is that the ability to make accurate predictions in sequence space is central to the ability to use reinforcement learning for model development -- this is related to the generation of synthetic data and is at the heart of some of the most powerful models currently -- such as the Alpha series by DeepMind. 

Combining these two ideas indicates that reinforcement learning is likely to be an intractable path for the development of virtual cell models -- at least to the extent that we want to model their state trajectories over time.   This point is not absolute -- the conclusion above does suggest that powerful computable predictors can exist -- but they must necessarily be very complex (i.e., very large networks) which require significant amounts of data to adequately train.

Many parallels have been drawn between the rapid advances in AI for language, images, and video to biological systems. It’s important to distinguish the domains and techniques that are used, however, to assess the validity of these claims.

1. Reinforcement learning (particularly combined with generative models) has been at the center of the most rapid advances in AI to date. Typically, this has taken the form of both self-play and sequence prediction both of which require reward estimators. These tools are very nascent in the life sciences.

2. Synthetic data has and will continue to be central to advancing AI models and is central to the development of simulators of virtual cells. I have written previously about synthetic data9 and its application in biology.

The important intuition is that tools that have been the most successful in current AI models are much less applicable in biological systems. It is unlikely that we will have the next sequence simulator model be “Alpha-Cell” using training methods of self-play, reinforcement learning, and synthetic data generation.

What about PINNs? (Physics Informed Neural Networks)

The computational limits above relate to arbitrary sequence predictors and the functions that generate them and more specifically the ability to generate arbitrarily complex sequence generators.  This is the analog to the "scaling transformer" approach or the agnostic algorithm, which from a theoretical perspective, at least as it relates to the matching availability of data to train, does not appear like an approach that will be successful in its basic form for developing cell simulators -- though some companies are pursuing this.     

However, physical systems, regardless of how complex, are not arbitrary sequences -- they do adhere to physics, even if we cannot explicitly calculate them.  The complexity of the system still matters in terms of the complexity of the generating algorithm, but we may consider that the domain that needs to be covered is still bounded, it is not arbitrarily large (though if one were to consider a situation like cancer, this assumption may not be valid).  It remains also the case that developing time-resolved simulations of cells will still require time-resolved data to train -- even a physics-informed network or neural operator needs data points to calculate predicted losses for the equations it generates.   It also remains the case that biological systems can be highly non-linear in their response dynamics -- for example, a single transcription factor or a SNP that introduces a codon variant that changes the reaction kinetics of protein can both significantly alter the future trajectory of a cell.   This will remain an outstanding challenge in developing fully robust models, however, I expect that substantial progress will be achievable in time.

An important limitation on the utility of PINNs (or their cousins PINOs — Physics Informed Neural Operators) is that their dependence on training data is related to the prior knowledge of their underlying physics. In basic terms, these models use physical inductive priors to constrain the search space that is explored during training. To the extent that these physical principles are well understood and already have underlying governing equations, this space is more constrained, and a network can be trained with fewer observational data points. This is often the case in real world systems governed by physics such as fluid flow, materials properties, etc. In biological systems, this context is not nearly as robust and the characteristics of the dynamics of interacting components is substantially more variable and less well understood. In practice, this means that the functional search space is much larger as and such the observational data required must also be larger for effective learning. For the interested reader, more information in PINNs and PINOs can be found at this reference10.

A great deal of the success of modeling cells will depend on the degree to which they are bounded. In the context of modeling cells from a perturbation perspective (which is the case for most drug development), the challenge will be to determine how such bounded cell states are bridged — like the Waddington landscape, it’s like pushing a ball up a hill, rolling down the next, and determining whether it will settle in a new valley.

Developing simulators that can predict this sequence of events is a considerable challenge, but one that is worth pursuing.

A Moment of Transparency

Every frontier AI lab is talking about "curing cancer" and the rapid advances in biotechnology and medicine that will be achievable, but it's important to take note -- the following is a quote from Dario Amodei, CEO of Anthropic:

... if I try and give an exact number it's just going to sound like hype but like a thing I could imagine is like I don't know like two to three years from now we have AI systems that are capable of making that kind of Discovery 5 years from now those discoveries are actually being made and 5 years after that it's all gone through the regulatory apparatus and really so you know we're talking about more we're talking about you know a little over a decade…

…but really I'm just pulling things out of my hat here like I don't know that much about drug discovery I don't know that much about biology, and frankly although I invented AI scaling I don't know that much about that either I can't predict it...11

To be fair, I am bullish on the ability of AI to make significant advances in biology, and I believe it is a good thing that frontier companies are addressing this potential, but we should take note when the leaders in this field tell you they don't actually know anything about it and adjust expectations accordingly.

There is significantly more to write on this topic, but it will be left for future articles.

If this article is interesting, please consider sharing and subscribing with the links below:

Share

For more see: jasonsteiner.xyz

References

1

Virtual Cells

Jason Steiner

·

May 24, 2024

Preamble and TLDR: This article is about virtual cells and the prospect of being able to simulate biology at the molecular, cellular, organ, and perhaps organism level. Like other articles in this Substack, it will be semi-technical, but largely oriented toward developing an intuition for the dimension of these challenges and the approaches for solution…

Read full story

9

To Be Or Not To Be Synthetic: Data In Bio x ML

Jason Steiner

·

February 23, 2024

Summary: Synthetic data is becoming more central to the training and development of deep neural networks. This article describes some of the intuition around its utility in different domains. If there is one thing that is coming to best characterize the ethos of deep learning it is that data is of paramount importance. Computational architectures are f…

Read full story