Recent progress inengineering highly promising biocatalystshasincreasingly involved machine learning methods. These methods leverageexisting experimental and simulation data to aid in the discoveryand annotation of promising enzymes, as well as in suggesting beneficialmutations for improving known targets. The field of machine learningfor protein engineering is gathering steam, driven by recent successstories and notable progress in other areas. It already encompassesambitious tasks such as understanding and predicting protein structureand function, catalytic efficiency, enantioselectivity, protein dynamics,stability, solubility, aggregation, and more. Nonetheless, the fieldis still evolving, with many challenges to overcome and questionsto address. In this Perspective, we provide an overview of ongoingtrends in this domain, highlight recent case studies, and examinethe current limitations of machine learning-based methods. We emphasizethe crucial importance of thorough experimental validation of emergingmodels before their use for rational protein design. We present ouropinions on the fundamental problems and outline the potential directionsfor future research. Its potential stems from its ability to mimicand harness the power of nature using cells and enzymes that haveevolved over millions of years to perform specific chemical reactionswith high efficiency. This makes it possible to transform chemicalcompounds selectively and efficiently, providing an alternative totraditional chemical catalysis, which often requires harsh conditionsand toxic chemicals.9 In the past,researchers would accumulate observations from individual experimentsand use the resulting data to formulate fundamental rules. They thencreated simulations based on these rules to better understand thesystem under investigation. As computational power has increased,researchers have been able to shift toward data-driven methods thatrely on machine learning , we start with a brief introduction to the topic. We will coverthe basics of ML-based pipelines and vocabulary, highlight similaritiesbetween protein engineering and other domains from a ML perspective,outline the main traits that distinguish protein data from other datatypes regularly used in ML, and summarize the challenges of performingML with protein data.2.1artificial intelligence (AI). Its primary purpose is to learn patterns directly from availabledata and use the learned patterns to generate predictions for newdata. Its main difference from other methods for modeling a system’sbehavior, such as quantum mechanical calculations, is that ML doesnot rely on hard-coded rules to make predictions. Instead, ML modelsare mathematical functions that depend on generic parameters whosevalues are obtained (learned) through optimization using availabledata and an optimization criterion, the so-called loss function.Machine learningis often seen as a subcategory of features. Features may be obtained by simple encoding of the raw data, e.g., end-to-end learning and one-hotencoding, but they may also represent more involved quantitiesderived from the raw data. For example, when predicting solubilityfrom a protein sequence, the features may be simple amino acid counts,propensities of different residues to form secondary structures, conservationscores of proteins, or variables representing aggregated physicochemicalproperties.26 Choosing informative anddiscriminating features that provide relevant information about theunderlying pattern in the data is crucial in ML because the featuresare the only data characteristics that the algorithm will exploitduring training and when making predictions based on future inputs.Since the final model is derived from the input data, carefuldata collection is vitally important for machine learning. In particular,any biases, measurement noise, and imbalances must be recognized andaccounted for. Moreover, as ML is based on mathematical functions,every data point in a data set typically needs to be represented asa vector of numbers that are commonly referred to as supervisedlearning problems, the goal is to predict a particular property(known as a label) for each data point  and regression problems involving labelscorresponding to numerical values . In contrast, the goal in unsupervised learning problems is to identify patterns in unlabeled data. Unsupervisedlearning techniques include clustering algorithms and data compressionor projection methods, such as principal component analysis  that forcethe model to learn a more compact way of representing the data andtheir distribution. The algorithms that aim to capture the data distributionto generate new samples belong to a class of ML models called generativemodels. The most recent examples of this class include diffusionmodels, which have recently been used to generate protein backbonestructures28 and predict the binding of aflexible ligand to a protein29 (X0) in a stepwise manner toultimately obtain a maximally noised sample (XT). The sequences of increasingly noisy data are then reversedand used to train a model to denoise each individual step by havingthe (less noisy) sample Xt-1 serve as a “label” for the (more noisy) followingsample Xt. For more details on diffusiongenerative models and their applications in bioinformatics, see therecent review.30 Alternatively, we canavoid the need for labels by masking a part of the input, e.g., a residue in a protein sequence or structure,33 and training a model that will predict the masked part. In otherwords, the original data  are treated as the label for the corresponding data point.Such approaches belong to the methods of self-supervised learning  and related metricssuch as the root-mean-square error (RMSE). More complex problems oftenrequire customized metrics. For instance, in a protein structure predictiontask, one might use the MSE between the predicted and actual locationsof Cα atoms, which can be expressed using a fixed coordinatesystem  or in terms of the local coordinates ofeach residue . Both metrics indicate how well a predictedstructure aligns with the corresponding actual structure.Thechoice of evaluation metric depends on the task at hand. Classificationproblems are mainly evaluated based on 2.2One of the strengthsof machine learning is its universality, as the algorithms used forprotein engineering tasks are similar to those used in other domains.Therefore, scientists working with protein data can reuse and buildupon existing solutions from other fields, such as natural languageprocessing, computer vision, and network analysis.e.g., to predict lexical categories or identifyrelevant information) or phrases . This data structure resembles that of annotated proteindata sets with labels representing protein stability, binding affinity,specificity, or other characteristics.34 Moreover, the complexities of the relationships between proteinsequence, structure, and function are reminiscent of those in humanlanguages, prompting researchers to adapt transformer-based large language models used in NLP to protein engineering tasks.35Natural languageprocessing (NLP) is a field of computer science that aims to teachcomputers how to understand and handle natural languages. New ML techniqueshave enabled great advances in this field in recent years. For instance,a common task in NLP is to generate semantically and grammaticallycorrect sentences. In protein engineering, the strings of amino acidsrepresenting primary sequences can be regarded as words constructedfrom an alphabet of twenty letters representing the canonical aminoacids. These words can represent secondary structures or other motifsthat can be combined in meaningful ways to create sentences in thelanguage of protein structure that correspond to functional proteins.Another common task in NLP is the assignment of labels to individualwords  via the application of a discretegrid. The resulting representations are similar to those of volumetric3D images, enabling the application of ML architectures, such as convolutionalneural networks, that were originally designed to process image data.These networks learn representations through convolution and hierarchicalaggregation and have recently been used to predict protein mutationlandscapes,42 can be applied to molecular dynamics trajectories to detectevents such as the opening of a tunnel.43 Video processing techniques can thus be adapted to clarify a protein’sfunction by analyzing the movement of individual atoms or groups ofatoms within a protein structure in a manner similar to the movementof objects or groups of objects in a video. Moreover, there have beenremarkable advances in ML techniques for video synthesis46 that could inspire new methods for capturing and synthesizing proteindynamics.The parallels between images and protein structures can befurtherexploited by adapting techniques developed for video analysis to predictprotein dynamics. For example, a trajectory generated in a moleculardynamics simulation can be regarded as a temporal sequence of 3D images.This makes protein dynamics analysis similar to video processing andimplies that video methods for event detection47 and protein–protein interaction networks.48 Interactions between proteins or between a proteinand a ligand can be represented as networks (graphs) in which nodescorrespond to proteins and ligands while edges correspond to biologicalrelationships between them. Once such a network has been defined,link prediction or community detection methods can be applied.49 Alternatively, a protein structure can be representedas a network where nodes correspond to residues or individual atomsand edges correspond to inter-residue interactions or interatomicbonds. This makes it possible to use graph-based ML algorithms fortasks such as predicting protein function, solubility, or toxicity.51 Moreover, data on protein interactomes and the structures of smallmolecules have been used to drive theoretical research on graph-basedML: the best-established benchmark for graph learning, OGB, includesmultiple biochemical data sets of this type.52Finally, one more domain is relevant to protein engineering:networkanalysis, which involves studying the properties and structures ofinterconnected elements. Network analysis techniques have been usedsuccessfully to study diverse social and biological networks, includingthe development of Covid-19-related sentiment on social networks2.3As discussed above, there are striking similaritiesbetween protein engineering tasks and other ML domains, includingnatural language processing, computer vision, and network analysis.However, protein data also present unique challenges related to therepresentation of proteins, the construction of labeled data sets,and the establishment of robust training protocols.The choiceof protein representation is a key step in all protein-related computationaltasks. Proteins can be represented at different levels of detail,from a discrete and accurate 1D representation of their amino acidsequence to a continuous and less accurate 3D representation of everyatom position, including or excluding chemical bonds. The selectedrepresentation determines the type and amount of information availableto the computational model and the range of applicable model architectures.nth character encodes the amino acid at the nth position in the protein sequence. In silico, the amino acids are typically represented using one-hot encoding.When the amount of data available for training is not enough for end-to-endlearning, one-hot encoding of sequences can further be transformedinto values corresponding to specific physicochemical characteristicsof amino acids, e.g., using AA indices.53 These indices provide additional informationand interpretability to the pipeline, although they were shown toperform on par with random vectors for some tasks.55 Another strategy to enrich the protein sequence representation isto include evolutionary information using a multiple sequencealignment (MSA) instead of a single protein sequence. Thisevolutionary information is valuable in various tasks, most notablyin structure prediction because the covariance of different residuepositions in the sequence can be related to the residues’ spatialproximity.56The go-to representation of a protein sequence is a string (word)constructed using an alphabet of 20 amino acids (letters). The lengthof the string equals the number of residues, and the e.g., all atoms except hydrogens), or justkey components of residues .Moreover, they may include different types of information about theseatoms and/or residues. Ideally, to enable data-efficient model training,structural representations should be invariant to rotation, translation,and reflection. However, straightforward representations based onthe 3D coordinates of the residues (atoms) lack this property. Itis therefore common to represent protein structure using an inter-residueor interatomic distance matrix in which each row and column is assignedto a specific residue (or atom) of the protein and the value of eachmatrix entry is equal to the distance between the corresponding residues(atoms). Such a matrix is necessarily symmetric; therefore, it iscommon to take the upper (or lower) triangular part and convert itinto a 1D vector for processing, e.g., by a neuralnetwork. While this representation is rotationally and translationallyinvariant, it is also inherently redundant, as its spatial complexityis quadratic with respect to the number of residues (atoms).The options for representinga protein’s structure are morevaried; representations may include all atoms, only selected chemicalelements  betweenthe nodes, or both.58 Graph-based proteinrepresentations are very flexible because the definition of the nodesand edges can be tailored to specific tasks and they can be made equivariant to rotation, translation, and reflection.59 A convenient definition of the nodes and edgescan also introduce an inductive bias that improves themodel performance. For example, edges corresponding to chemical bondscan guide a model toward learning chemical knowledge more rapidlyor with less data than would otherwise be needed. Graph neural network(GNN) architectures have recently achieved state-of-the-art performancein multiple protein-related tasks, as exemplified by the DeepFRI60 and HIGH-PPI61 methodsfor predicting protein function and protein–protein interactions,respectively. Special graph-based protein representations, such aspoint clouds (no edges) or complete graphs (a full set of edges),are especially convenient for processing using powerful transformermodels.63Graph-based protein representations have recently attracted considerableattention.64 Itsaim is to remove the suboptimality of human-made choices by inferringthe representation parameters from existing data. Furthermore, itis often possible to learn such a representation through self-supervisedtraining, i.e., without the need for annotated data.These representations can be obtained from the sequence data, e.g., as done by the ESM language models,67 as well as from large structural data sets, e.g., as done by GearNet.68 Moreand more models combine both sources of data, e.g., ESM-GearNet.32A more general approach to proteinrepresentation is to directlylearn the representation by a deep learning model, a direction thatis currently on the rise in biology.69 and are thus generallyinsufficient for representative sampling of vast mutational spaces.Conversely, high-throughput methods such as Deep Mutational Scanning(DMS) are prone to data quality issues69 and face a throughput bottleneck in the case of enzymes whose screeningis considerably slower than sequencing.70Obtaining appropriatelylabeled data sets can be challenging whenseeking to apply ML in enzyme engineering because there is often atrade-off between data quality and quantity when selecting methodsfor acquiring experimental data. Most reliable biochemical methodsusing purpose-built instruments can only provide data for small numbersof protein variants71 It is important to consider these biases duringdata set construction and when interpreting model outputs since thecomposition of the training set significantly affects the space ofpatterns explored by the model.The process of compiling new data sets from multiple sourcescanbe complicated by the inconsistency of conventions in protein research.These inconsistencies include the differing biases of various experimentaltools, the use of different distributions for data normalization,and inconsistent definitions of quantities such as stability, solubility,and enzyme activity. All of these can introduce errors into constructeddata sets, for example, by causing contradictory labels to be appliedto the same protein. Protein data may also contain biases introducedby the design strategy. For example, alanine tends to be overrepresentedin mutational data due to the widely used alanine scanning technique.72 GPT-4,73 or BioGPT74 are remarkably effective at summarizing text on large scales,mining relevant data from publications still requires considerablehuman effort.A significant amount of proteindata has been gathered over theyears. However, much of these data are proprietary and thus inaccessibleto the academic community. In addition, publicly available data setsare often published in an unstructured way, which limits their usability.While large language models such as GPT-3,e.g., TorchProtein,75 offer preprocessed data sets for various proteinscience tasks, making protein research more accessible to ML expertsfrom other domains. Other packages, such as PyPEF76 provide frameworks for the integration of simpler ML modelstogether with special encodings derived from the AAindex databaseof physicochemical and biochemical properties of amino acids.53 Despite this progress, the development of MLmodels for protein data requires a certain level of biochemistry domainknowledge to account for the specifics of protein data absent in otherapplication areas of ML. These specifics include the evolutionaryrelationships and structural similarities between proteins. Modelsproduced without the benefit of such expertise may lack practicalutility due to data handling errors.Some ML packages, 77 It may also be important to consider multiplelevels of separation when dealing with protein data, e.g., consisting of mutations and their effects. For example, one mightwant to ensure that the same substitutions, positions, or proteinsdo not appear in the training and test sets. Defining protein similarityis also a challenging task for which multiple strategies exist. Manyof these strategies involve clustering proteins based on sequenceidentity or similarity thresholds and then ensuring that all membersof a given cluster are assigned to the same set when splitting thedata. This strategy is particularly useful for constructing labeleddata sets of protein structures because such data sets are primarilysourced from large redundant databases such as PDB78 (see Table S2). However, clusteringin the sequence space can be insufficient in some cases. For example,distantly related proteins may have very similar active site geometrieseven though their sequence homology is low.79 Clustering may thus be performed at the level of the structuralrepresentation instead. While such strategies have only rarely beenused in the past, they may become more common due to the emergenceof new tools for protein structure searching such as Foldseek.80One common type of datahandling error is data leakage betweendata splits. The training, validation, and test sets should not sharethe same (or nearly the same) data points because such overlaps couldlead to over estimating the model’s performance on new data,compromising the model’s evaluation. In some data sets, allof the data points are distinct enough that randomly splitting theavailable data into disjoint sets is a viable strategy. However, morecomplex splitting strategies are often needed when dealing with proteindata to avoid problems such as evolutionary data leakage.3in silico prediction is necessaryto keep pace with the exponentially growing number of deposited sequences.We will then cover the available labeled data sets and state-of-the-artML tools for predicting mutational effects in proteins, as well asstrategies for protein design based on their predictions. In addition,we will review published methods for leveraging unlabeled proteinsequence and structure data sets to help guide protein engineering.Finally, we conclude with examples showing how ML models of proteindynamics can facilitate the selection of promising mutations.This section provides a brief overview of the proteinengineeringtasks that have already drawn the attention of machine learning developers.The first of these tasks is functional annotation, which is importantbecause the overwhelming majority of sequences in protein databasesremain unannotated and 3.183 Likewise, in biocatalysis, one needs to know thefunction of enzymes to assemble a biosynthetic pathway from plausibleenzymatic reactions.85Knowledge of proteinfunctions is fundamental for protein engineeringpipelines. For instance, in protein fitness optimization, scientistsstart from a characterized wild-type sequence with some degree ofthe desired function.Table S2) contains 2.5 billion sequences to date.Functional annotation is particularly important for enzymes. A broad-levelannotation  can be achievedrelatively easily by sequence homology and by searching for proteindomain motifs, but detailed annotation of enzyme substrates and productscurrently requires experimental characterization. To accelerate thisprocess, a substantial recent effort has been dedicated to the developmentof novel computational methods for functional annotation.Traditionally, scientistshave characterized the functions of proteins via laborious, time-consuming,and costly wet lab experiments. However, owing to high-throughputDNA sequencing, the exponentially growing repertoire of protein sequencesis reaching numbers far beyond the capabilities of experimental functionalannotation; for example, the Big Fantastic Database  numbers, the most commonlyused nomenclature for enzymatic functions. EC numbers are a hierarchicalclassification system categorizing enzymatic reactions at four levelsof detail, with the fourth level being the most detailed. An EC numbergroups together proteins with the same enzymatic activity regardlessof the reaction mechanisms.96 The datafrom the above-listed databases have also been post-processed andorganized into data sets such as ECREACT97 or EnzymeMap,98 which should furtherfacilitate the development of computational models.Thedeveloped computational methods heavily rely on data sets ofpreviously characterized enzymatic functions for training. For example,information on protein families and domains is often sourced fromthe Pfam,99 is still to be explored moreby the community. The outputs of these models generally fall intothree categories based on the resolution of predictions. First, themost general models predict protein families and domains. Second,the EC class-predictive models provide a more detailed estimationof enzymatic activity. Finally, the most comprehensive picture ofenzymatic activity requires models for predicting an enzyme’ssubstrates and corresponding products. Several recent deep learningmodels predict protein families and domains.101 Although such models are crucial for studying proteins, they haveonly a limited applicability in enzymatic activity prediction, asa single protein family can combine enzymes catalyzing different reactions.102 To meet the needs of protein engineering, themodels predicting the EC numbers appear to be more relevant, as theycan capture the catalytic activities.The modelsfor enzymatic activity prediction typically use enzymeamino acid sequences as input, as the goal is to directly annotatethe outputs of high-throughput DNA sequencing. The incorporation ofthe structural inputs, facilitated by the recent breakthroughs instructure prediction,105k-nearest neighbor-based classifiers,108 support vector machines (SVM),112 random forests,114 and deep learning.119 Most methods approached the prediction of EC numbers as a classificationproblem, which led to poor performance on the under-represented ECcategories. The recent deep-learning-based method119 tackled the EC number prediction via a contrastive learningapproach, training a Siamese neural network on top of sequence embeddings from a pretrained protein language model.66 The resulting predictive algorithm, CLEAN, canbetter identify enzyme sequences that belong to any EC category, includingunderrepresented ones. CLEAN achieved state-of-the-art performancein EC number prediction in silico, and it was experimentallyvalidated in vitro using high-performance liquidchromatography–mass spectrometry coupled with enzyme kineticanalysis on a set of previously misannotated halogenase enzyme sequences.Over the years, the communityattempted to predict EC numbers usingmultiple sequence alignment and position-specific scoring matrices(PSSM) or hidden Markov model (HMM) profiles,85 For instance, planning of biosynthesis has beentackled by predicting the chemical structure of the substrate andthe required enzyme EC number from the provided enzymatic productusing a transformer-based neural network.97 Several other published deep learning models aspired to estimatethe substrate of an enzymatic reaction based on its product.121 Such models can be used to prioritize enzyme selection based onthe EC class or the substrate/product pair. However, the assignmentof a specific enzyme sequence  to a desired reaction remains a challenge for future development.EC class prediction enables downstream applications such as retrobiosynthesis.123 These DL-based models take pairs of a protein sequence and a smallmolecule as inputs and predict their possible interaction. Unfortunately,none of the general substrate–enzyme interaction models werevalidated in wet lab experiments. Furthermore, in Kroll et al. theauthors admit poor generalization of the model to out-of-sample substrates.123 Moreover, enzyme-family-specific models wereshown to outperform the general models in predicting the enzyme–substrateinteractions.124 To sum up, the practicalapplicability of general enzyme–substrate interaction modelshas yet to be determined.Recently, the first general models predicting interaction betweenindividual enzymes and substrates/products were published.3.2e.g., to predicta solubility score, binding energy, or melting temperature of a givenprotein), applying the trained model to independently predict labelsof the reference protein and its variant, and then taking the differencebetween the two predicted scores. The attractiveness of this strategycomes from large annotated data sets of wild-type proteins availablefor training. For example, the Protein Structure Initiative125 generated a massive data set TargetTrack, oftenused for protein solubility prediction.26 Additionally, the more recent Meltome Atlas of protein stabilityobtained by liquid chromatography-tandem mass spectrometry126 was used for predicting melting temperatures.127 Our ongoing effort to predict highly valuablemelting temperatures solely from the protein sequence resulted inthe development of the TmProt software tool (https://loschmidt.chemi.muni.cz/tmprot/). Large labeled data sets usually provide enough training data forpowerful end-to-end deep learning.129 Nonetheless, when the training data set does notcontain mutations, a few substitutions will usually result in similarpredicted labels , in contrastto dramatic changes often observed in experiments. Therefore, thestrategy of taking the difference between the predicted labels fora reference protein and its variant typically fails to produce reliablepredictors for mutational effects.130The ability to predict mutational effects onvarious protein properties, such as solubility, stability, aggregation,function, and enantioselectivity, is another desirable goal of proteinengineering. From the machine learning perspective, this implies havinga model that takes a reference protein and its variant as the inputand predicts the change in the studied property as the output. Intuitively,this could be achieved by supervised learning on the labeled datasets of wild-type proteins first  and biased toward several overrepresentedproteins.ad hoc becausethey are generally complementary to a study’s main results.132 This highlights the importance of establishingand maintaining databases with protein annotations to facilitate datadiscoverability and reuse. For example, we recently released SoluProtMutDB,133 which currently has almost 33 000 labeledentries concerning mutational effects on the solubility and expressionof over 100 proteins. This database incorporates all of the data pointsthat were recently used to develop solubility predictors  or the Gibbs free energy difference between the foldedand unfolded states (ΔΔG). Several proteinstability databases exist, including FireProtDB,158 ThermoMutDB,138 and ProThermDB,135 and their data have often been used to trainML predictors that have achieved Pearson’s correlations ofup to 0.6 and RMSE values of 1.5 kcal/mol when applied to independenttest sets.159 Interestingly, these numbershave barely changed over the past decade, indicating that a qualitativeparadigm shift might be needed to advance ML-based prediction of mutation-inducedprotein stability changes.132 It is possiblethat large new data sets could provide the necessary boost, and someexciting studies collecting such data are already appearing: cDNAdisplay proteolysis was recently used to measure the thermodynamicstability of around 850 000 single-point and selected double-pointmutants of 354 natural and 188 de novo designed proteindomains between 40 and 72 amino acids in length.140Protein stability measurements are another type ofwidely availablebiophysical data that can be used in ML. Protein stability is typicallyquantified in terms of the melting temperature  data sets for benchmarkingeffect predictors,164 and we expect this trend to continue as more data sets appear. Notableworks of this type include the recently published activity landscapesof the phosphatase,165 dihydrofolate reductase,166 DNA polymerase,167 and palmitoylethanolamide transferase.168Changes in catalytic activity upon mutation alsoattract the attentionof ML researchers. Predicting mutational effects on enzyme activityis more challenging than predicting protein stability and solubilitydue to the enormous diversity of enzymatic mechanisms. One rich sourceof such mutational data is large-scale deep mutational scanning.169 It contains data derivedfrom UniProt annotations representing over 90 000 mutationaleffects in 2130 enzymes. However, its potential in ML has not yetbeen explored.DMS can be applied to a wide range of enzyme functions dueto itshigh flexibility with respect to selection procedures. However, itshigh throughput comes at the cost of limiting the number of proteintargets that can be used in a study; often, only a single case isexamined. The desire to target multiple enzymes simultaneously motivatedthe creation of another notable database of enzyme activity changes:D3DistalMutation.149 enzyme enantioselectivity,171 kinetic constants,144 temperature sensitivity,172 or temperatureoptima.148 In addition, several mutational data setsthat can be used for ML-based tools focusing on protein folding rates,binding, and aggregation have been deposited in the VariBench benchmarkdata set.173 Selected recent examples ofthese tools are listed in Table S2.Other protein characteristics may also be usedas targets for proteinengineering and machine learning. These targets are often selectedbased on the enzyme of interest and may include important functionaltraits such as substrate specificity,3.3Whiletools for predicting effects of mutations have become increasinglyadvanced in recent years, in their simple form, they can only providelabels for a given substitution. However, the desired outcome of proteinengineering pipelines is to have a list of promising protein variantsfor experimental validation. Therefore, even if a reliable ML-basedtool for predicting effects of substitutions is available, the problemof suggesting promising hypothetical designs must be addressed. Thisproblem may become a major bottleneck, since even if the predictionof single- or multiple-point mutational effects is fast, evaluatingall possible combinations of mutations remains unfeasible. Therefore,there is a growing need for tools that simultaneously predict theeffects of mutations and reduce the search space, which is the focusof this subsection.174 This common nonadditive behavior, which is known as epistasis, hasprompted the development of ML models and combinatorial optimizationalgorithms capable of scoring or searching multipoint mutants by design.Several approaches discussed below have been proposed to overcomethis challenge, more on this topic can be found in a review.175A major challenge in the development ofsuch tools is finding ways to efficiently reduce the space of multipointmutants. In an analysis of nine case studies, Milton and coauthorsfound that the effects of half of the multipoint mutations influencingenzymatic properties could not be predicted using knowledge of thecorresponding single-point mutations, with the associated complexitiesresulting from direct interactions between residues in some casesand long-range interactions in others.176 For example, HotSpot Wizard 3.0177 achieves robust selection of hotspots by using a number of sequenceand structure-based filters to identify mutable residues for whichmutation effects are then quantified using the well-established Rosettaand FoldX tools. Another example is the FuncLib web server, whichcomputes promising single-point active-site mutations using evolutionaryconservation analysis and Rosetta-based stability calculations.178 This tool exhaustively models each combinationof mutants and ranks them by energy. Evolutionary information canalso be captured by ML-based models179 andused to suggest promising substitutions, e.g., aminoacids with conditional likelihoods higher than the wild-type.180One such approach is to produce a libraryof variants for screeningusing reliable physics- and evolution-based tools. Even a time-consumingpreselection of promising hotspots can drastically reduce the computationaltime of the downstream ML scoring and search.in silico by iteratively producingnew designs based on available ML scoring data  of the input thatalter the corresponding features in the desired direction. White-boxmethods are discussed further in the context of explainableAI in Mathematical optimization methods cangenerate promising proteinsequence candidates ing data 3. One grin silico optimization methodsenable theiterative generation of promising designs, they rely entirely on theability of a predictor to correctly score any given point in a mutationallandscape, which might be an unrealistically strong assumption. Apossible alternative  is to directly incorporate experimental validation intothe optimization loop. This experimental input can guide search algorithmsto more promising parts of the mutational landscape in a manner thatis akin to directed evolution. While advanced search methods of thistype have previously been used to improve traditional directed evolution,182 they do not fully exploit the potential of experimentally characterizingintermediate variants. ML-based active learning methodsaccelerate directed evolution by iteratively extracting knowledgefrom all characterized variants and selecting the most promising ones.185 By relying on the new experimental data, onlya limited number of training samples can be expected,186 confining the choice of ML models to thosewith a lower number of parameters, such as multilayer perceptrons (MLP) with as few as two layers.187 Arecent advance in the area of active learning is the development ofGFlowNets,188 networks designed to suggestdiverse and accurate candidates in a machine–expert loop toaccelerate scientific discovery. Studies using such networks havedemonstrated their potential for designing small molecules;189 however, the utility for the design of proteinsremains to be reliably demonstrated.While in silico techniques but also on combining multiple in silico approaches and sometimes different ML techniques.For example, focused training ML-assisted directed evolution (ftMLDE)186 combines unsupervised and supervised trainingapproaches by using unsupervised clustering to construct a trainingset for supervised classifiers. These classifiers are then used toselect promising mutants with tools such as CLADE 2.0.190 Similarly, unsupervised learning on millionsof sequences was used to obtain a protein representation called UniRep,191 which was further tuned in an unsupervisedmanner to obtained eUniRep (evotuned UniRep) for proteins relatedto the target sequence, leading to an informative set of featuresfor proteins in general as well as for the target in particular. Suchrepresentation enabled data-efficient supervised learning of a modelfor guiding in silico evolution.192 Alternatively, an unsupervised “probability densitymodel” has been used to produce an “evolutionary densityscore”, a feature that was then used to augment a small numberof labeled data points on which a light model was trained in a supervisedmanner. Interestingly, such an approach was shown to outperform thesupervised fine-tuning of the probability density modelpretrained in an unsupervised manner.193 Another approach194 combines self-supervisedlarge protein language models with a supervised structure-to-sequencepredictor in a new and more general framework called LM-design thatis claimed to advance the state of the art in predicting a proteinsequence corresponding to a starting backbone structure, sometimescalled “inverse folding”. While inverse folding doesnot explicitly search the mutational landscape, it can be used toidentify promising mutations by inputting an existing protein structureand a partially masked sequence and using the inverse folding toolto propose amino acids for the masked parts.Powerful design techniquesfor the exploration of variants of enzymesand other proteins often rely not only on combining experimental and 3.4large language models (LLM)have become popular tools for solving NLP problems ranging from languagetranslation to sentiment analysis.195 Thismajor paradigm shift was driven by the realization that even unlabeleddata contain useful information: the distributional hypothesis suggeststhat the meaning of words can be deduced by analyzing how often andwith what partner words they appear in various texts.196 Analogously, in biology, we can regard proteinsas sequences based on “the grammar of the language of life”,which implies that the distribution of amino acids at specific locationscan provide valuable insights that could be used to help predict theeffects of substitutions on protein function, thereby reducing thereliance on external data sources.66 Forexample, Elnaggar et al. showed that the embeddings generated by aLLM, when used as input features, can effectively facilitate the developmentof small supervised models whose predictive power rivals that of state-of-the-artmethods relying on evolutionary information obtained from MSAs.197 Additionally, the protein language model ESM-2was recently trained on protein sequences from the UniRef databaseto predict 15% of masked amino acids in a given sequence.67 This made it possible to directly leverage sequenceinformation to greatly improve B-cell epitope prediction198 without a supplementary MSA. The inherent attentionmechanism of ESM-2 can also be used to facilitate protein structureprediction.67 We provide more examplesin 65 ESM-2,67 ProtT5,197 and ProtTrans,199 have become a popular way of representing sequential data, makingpretrained ESM models a frequent subject for transfer learning. The transfer of knowledge from these models has been tackled byfine-tuning the pretrained weights,200 by trainingmodels solely on top of the learned embeddings (keeping the pretrainedweights fixed),197 and also by introducingadapter modules201 between the trainedlayers for parameter-efficient fine-tuning.202 The power of the learned embeddings can also be witnessed in a completelyunsupervised setting. More specifically, the ESM-1v66 model was demonstrated to accurately score protein variantsby relying solely on the wild-type amino acid probabilities learnedby the protein language model during pretraining without any subsequentfine-tuning.Over the past decade, e.g., for identifying functionally importantconserved regions, insertions, or deletions to clarify the mechanismsdriving sequence divergence.Large deep-learning models are impressively capableof learninggeneral protein properties. For the cases in which a detailed understandingof a specific protein or protein family is required rather than generalpatterns that hold across different protein families, sequence-basedmodels that can learn distributional patterns from a single MSA areavailable. MSAs have already proven to be a rich source of evolutionaryinformation, 66 For example, excellent protein structure predictionperformance was achieved by combining MSA inputs with an advancedtransformer architecture.216 In addition, Xie et al. used the maximum entropy (MaxEnt) principleto infer statistical energy from homologous sequences143 and found that the inferred statistical energyof active site residues correlated significantly with enzyme activity,whereas that of residues distant from the active site correlated withprotein stability. Hsu et al. observed that a hybrid linear regressionmodel combining the evolutionary density score from MSA-based ML modelswith labeled one-hot encoded protein sequence data sets demonstratedsuperior performance in a range of protein fitness prediction tasks,even for the sizes of the labeled data sets in the range of 48–240data points.193 A similar effect for smalldata sets of 50–250 data points was reported by Illig et al.217 Ding et al. showed that variational autoencoderscan capture phylogenetic relationships in the geometry of the latentspace of an MSA and further demonstrated that the free energies ofsequences assigned by the latent model can be used to predict mutation-inducedchanges in stability.218 Building on thesefindings, the geometric structure of a latent space was recently usedto guide the design of a haloalkane dehalogenase.219 In addition, by exploring the latent manifold underlyingthe sequence information, we can uncover dependencies that may notbe readily apparent in the raw latent space embeddings.220Analyses of evolutionary data andcomparisons of assigned probabilitiesfor mutant and wild-type sequences have also shown that ML-based modelscan predict the effects of mutations in deep mutational scanning experimentsmore accurately than established methods based on evolutionary data.Despite its advantages, MSAs also havesome drawbacks. First,it can be difficult to create an MSA that contains enough evolutionarilyrelevant sequences to establish strong patterns at key amino acidpositions. Second, the creation of an MSA is often seen as somethingof a craft rather than a systematic procedure, and the alignment processcan be sensitive to user choices, including the choice of substitutionmatrices, gap penalties, and the number of iterations. Poor choicesmay produce an MSA with improperly aligned residues because too fewiterations were performed during the alignment process or becausea significant number of gaps were introduced , necessitating additional preprocessingand trimming of the MSA to achieve optimal results. Moreover, aligningnew sequences to an MSA can be a challenging task that requires carefulconsideration of the alignment parameters and sequence properties.Despite these challenges, MSAs remain a powerful source of informationfor studying the relationships and evolutionary history of biologicalsequences.213 After fine-tuning the model on the phenylalaninehydroxylase (PAH) sequence family, ProtT-VAE was used to predict variantswith up to 100 mutations, resulting in a 2.5× increase in catalyticactivity over the human wild-type. Similarly, Luo et al. combineda pretrained language model with an MSA-specific direct coupling methodto capture both general protein syntax and protein-specific epistasis.214 The resulting model, ECNet, was used to engineerTEM-1 β-lactamase variants with improved ampicillin resistance.Recent studies have also explored the opportunitiesprovided byexamining patterns at two contrasting scales within the protein sequencespace: general patterns in the large universe of protein sequencesand local distributions of sequence patterns in specific protein families.These works have combined large pretrained models with lightweight,easily retrainable components for efficient family-specific adaptation.For example, Sevgen et al. designed ProtT-VAE, a fusion of a transformerpretrained on 46 million UniRef50 sequences and an autoencoder thatenables fine-tuning on a sequence library of interest.221 or structural similarityprediction for protein sequences.222 Thesetwo methods employ pseudolabeling or a custom similarity metric toenable a supervised learning task to profit from large initially unlabeleddata. In some cases, even self-supervised methods are viewed as semisupervised223 in that they learn powerful representationsfrom unlabeled data that can then be fine-tuned using small labeleddata sets. However, the difference from semisupervised methods isthat self-supervised methods build on the established methodologyof supervised learning as they formulate a supervised proxy task usingsynthetic labels generated automatically from the large unlabeleddata set. Such self-supervised methods are powering the recent successesof protein language models, discussed at the beginning of this section,and appear to be generally more successful than traditional semisupervisedmethods.The utilization of unlabeled data in a semisupervised setting hasbeen tried, for example, in the context of secondary structure predictionfor orphan sequences31 Additionally, Zhang et al. designed multiple general tasks for theself-supervised pretraining of graph neural networks for protein structures,leading to improved performance in various downstream tasks.68 Self-supervised learning on protein structureshas also been used to suggest protein sequences for specific backbonesand generate de novo protein structures.207In parallel to the explosive growth of sequence databases,theProtein Data Bank reached the landmark value of 200 000 entriesin April of 2023, providing a rich source of experimental structuraldata for self-supervised learning about protein structures. A numberof deep learning models have been fitted to large subsets of the PDBdata to leverage the natural diversity of the protein structures.For example, MutCompute was trained on a nonredundant sample of 19Kproteins to predict artificially masked residues based on the local3D environment, enabling the model to successfully capture phenotypiclandscapes associated with protein stability.3.5224 One way to capture a protein’s dynamicsis by studying its structural ensembles, which are available fromexisting protein structure databases in many cases. It has thereforebeen suggested that structure-based ML methods trained on such data, e.g., AlphaFold2, could provide insights into protein dynamics.225 Building on this hypothesis, Brotzakis andcoauthors proposed a reweighting procedure using AlphaFold2 predictionsand the FoldingDiff framework27 to generatestructural ensembles for disordered peptides.226 Such procedures could be useful alternatives to computationallyexpensive simulations in some cases. However, they are unlikely tobe useful for comparing structural ensembles of closely related proteinvariants because AlphaFold2 predictions, which rely on evolutionaryinformation obtained from MSAs, appear to be insensitive to single-pointmutations139 and so may be unable to accuratelycapture the often subtle differences between closely related proteinvariants.The preceding sectionsfocus on the analysis of static data. However,proteins are complex and dynamic systems, and their conformationalchanges and motions often provide key insights into reaction mechanismsthat cannot be obtained by studying static structures alone.227 In most cases, however,finding so-called collective variables (CVs) that effectively describethe system’s dynamic behavior is not straightforward. It hasbeen suggested that unsupervised learning methods could help solvethe problem of identifying CVs because they can learn from raw MDtrajectories without making assumptions about the CVs that are beingsought. In general, such methods try to find low-dimensional representationsthat are rich enough to reconstruct the original (or time-shifted)input.228 The resulting low-dimensionalprojections can then be used to build a simplified model of the protein’sdynamics. This is often done using Markov state models (MSMs), whichcan cluster the conformational space of the simulated molecule intoa tractable number of clusters (states). MSMs assume that the transitionsbetween these states are Markovian, i.e., the statein the next time step depends only on the current state regardlessof the previous trajectory. This approach was applied in conjunctionwith the end-to-end deep learning model VAMPnet, which was used tofind an optimal projection for the system under study.229 More recently, the CoVAMPnet framework230 was created by expanding on the VAMPnet approachto add interpretability capabilities and a method for MSM alignmentthat facilitates comparisons of MSMs for two sets of simulations.If applied to different variants of a given protein, CoVAMPnet couldpotentially be used to evaluate the effects of discriminative mutationson protein dynamics. However, this application is yet to be explored.Another option is to study molecular dynamics (MD)trajectories. MD data are obtained by performing simulations thatapply physical laws to calculate the future position of every atomin a protein after a given time step based on its 3D structure atthe current point in time. The resulting trajectories consist of aseries of snapshots capturing the protein configurations at successivepoints in time. Because each snapshot contains the coordinates ofall atoms in the simulated system, the simulation provides a largeamount of data, even for an average-sized protein. Where prior systemknowledge is available, one can use a coarse-grained model to reducethe system’s dimensionality and potentially capture major motionsof interest.231 in which supervised autoencoders were trainedon MD trajectories to identify the most significant differences inpairwise residue distances between protein variants. Another approachfor comparing dynamical changes in variant trajectories involves directlyanalyzing the distribution of low-dimensional projections of configurationsobtained by variational autoencoders. In this approach, similar spatialconfigurations tend to cluster in certain subspaces of the learnedlow-dimensional space. Such clustering can be used, for example, toanalyze the MD simulations of catalytic sites in the presence of differentsubstrates to identify structural differences that drive substratepreferences.232 Nonetheless, despite thosepromising studies, the problem of comparing MD trajectories of differentprotein variants in a systematic way remains largely unexplored, offeringan intriguing future direction for machine learning applications.In general, identifying dynamic features associated with biochemicaldifferences between protein variants is much more challenging thanfinding a low-dimensional representation for a single protein. Oneway to address this problem was demonstrated by the DiffNets framework,4Table S1.While the number of publications on ML for protein engineeringis growing rapidly 1, only a4.1Escherichia coli in minimal media.233 Ten of them were characterized in more detail and shown to haveexpression and catalytic parameters similar to those of the previouslycharacterized enzymes. In another case, Repecka et al. used generativeadversarial networks to create artificial malate dehydrogenases.212 Sixty of the resulting artificial proteins were tested experimentally, revealing that 13 had catalyticactivity comparable to that of natural enzymes. Another interestingcase study was reported by Madani et al., who used a large 1.2 billion-parameterlanguage model called ProGen for the conditional generation of 100artificial sequences that were fine-tuned to five distinct lysozymefamilies.35 Again, the artificial designshad expression levels and catalytic efficiencies similar to thoseof natural lysozymes even though their sequence identities with thenatural proteins were as low as 31.4% in some cases. In all threestudies, the authors could approach the expression and activity ofnatural sequences, but surpassing them by a significant margin stillremains a challenge.Severalpublications have showcased recent advances in the use of sequence-basedmodels, including both models trained on large sequence databasesand models trained on specific enzyme subfamilies. For example, Russet al. used a statistical model based on direct coupling analysisto learn the natural distribution of amino acids in chorismate mutasesand generate 1618 artificial enzyme sequences. Experimental studiesshowed that 30% of these proteins rescued the growth of enzyme-deficient 4.2234 Despite extensive research on enzymatic PETdepolymerization, most known PET-hydrolyzing enzymes have poor activityand require high temperatures or highly processed substrates to bepractically useful. Traditional protein engineering strategies haveimproved the thermostability and catalytic activity of PETase variantsunder certain conditions, but these variants still show low activityat mild temperatures. Lu et al. therefore tried to use structure-baseddeep learning to solve this problem.234 For this purpose, they used the MutCompute31 algorithm, which was trained to predict masked amino acids basedon their local 3D microenvironment in a crystal structure, to identifythe positions where the predicted probabilities of the wild-type aminoacids were comparatively low, suggesting that some other amino acidsare better “suited” to the corresponding structuralmicroenvironment. Eight of the top ten suggested substitutions producedsingle-point mutants with improved thermostability and activity. Combinatorialassembly of these substitutions yielded PETase variants with meltingtemperatures of up to 83.5 °C  and up to 38×the activity of the template enzymes at 50 °C.Designssuperior to wild-type templates could potentially be created by leveragingstructural data. One promising strategy for this purpose is to usethe local structural environment to identify positions suitable foroptimization in wild-type proteins. An insightful recent study onplastic-degrading enzymes demonstrated the power of this approach.235 Moreover, variants combining these substitutionswere more robust to purification and exhibited even greater thermotoleranceand thermostability. MutCompute predictions are orthogonal to force-fieldcalculations and phylogenetic analyses conducted with the popularautomated web tools PROSS236 and FireProt,237 and these predictions can be exploited to removedestabilizing substitutions from multiple-point mutants designed usingthose tools. This was recently demonstrated by the successful stabilizationof haloalkane dehalogenase DhaA115 .207 For experimentalvalidation, the authors targeted proteins that had been generatedby deep network hallucination in a previous study239 but proved to be mostly insoluble when expressed in E. coli. Ninety-six designs were processed using ProteinMPNN,of which 73 were soluble when expressed and 50 had the desired monomericor oligomeric state. Moreover, some maintained this state even at95 °C. This is a promising result because protein insolubilityappears to be a persistent problem with ML-based generated sequences.240 It will therefore be interesting to see ifmore enzyme sequences can be reengineered for improved solubilityin this way.An alternative to replacing specific residues to better match structuralpatterns in training data is to predict complete protein sequencescorresponding to a given backbone conformation from scratch. Exemplifyingthis approach, Dauparas et al. trained the graph-based neural networkProteinMPNN on a set of 19 700 high-resolution single-chainstructures from PDB and showed that this algorithm can rescue previouslyfailed designs by suggesting optimized protein sequences for givenscaffold templates.26 and integrated itinto EnzymeMiner (https://loschmidt.chemi.muni.cz/enzymeminer/), a more general pipeline for discovering enzymes with a desiredcatalytic activity that is available as a fully automated web service.This pipeline was recently used to mine promising industrially relevanthaloalkane dehalogenases241 and fluorinases.242 In both cases, we obtained soluble and activeenzymes with catalytic performance superior to that of previouslycharacterized wild-type enzymes and engineered biocatalysts. The broadapplicability of the SoluProt and EnzymeMiner web services is demonstratedby the fact that they have completed over 30 000 and 3000 jobs, respectively,in the two years that they have been online. An alternative to naturalsequence mining with EnzymeMiner is to use the deep neural networkProteinMPNN, which has been applied successfully in the denovo design of artificial luciferases based on computationallydesigned binding pockets.243 Functionalconstraints have also recently been incorporated into structure-basedgenerative models including diffusion-based deep learning methodsfor functional motif scaffolding.28At Loschmidt Laboratories, we developed the sequence-basedsolubilitypredictor SoluProt4.3The preceding examplesdemonstrate the potential of using unlabeleddata sets to suggest novel protein designs. Moreover, in in silico docking, Büchler et al. chose three criticalamino acid positions in an iron/α-ketoglutarate-dependent halogenasefor full randomization in a library targeted for the use in an algorithm-aidedenzyme engineering strategy.244 After collectingthe activity measurements for 504 unique variants, the authors traineda Gaussian-process-based model to explore in silico the remaining protein landscape for activity and selectivity. Thesubsequent experimental validation revealed active and selective halogenasevariants with over a 90-fold increase in the apparent kcat and a 300-fold increase in the total turnover number.A smaller data set of 80 variants of Sortase A was used by Saito etal. to train an ML model to score all possible variants for five mutatedpositions.245 After designing primer sequencesto include the top 50 variants in the second-round library, the authorsobserved most of the new variants showed high expression levels, withseveral demonstrating higher enzyme activity than the first-roundvariants. Reiterating the workflow, the authors constructed and validatedthe third-round library, again leading to a set of improved variants.Interestingly, the authors tested and stressed the importance of includingpoor-performing variants in the training data, and they still gotpromising results in the scenario in which the top-performing variantwas excluded from training. Even a smaller initial library of 20 chimerasequences was used in a different study by Greenhalgh et al. as astarting point for ten rounds of sequence optimization of alcohol-formingfatty acyl reductases, leading to an over twofold increase in fattyalcohol production compared to the starting sequences.246Several case studies combining machine learningand directed evolution have appeared recently. Based on the initial 247 This study drew on an earlier indel (insertions and deletions) mutagenesisexperiment targeting a reconstructed ancestral protein and aimed toidentify the factors responsible for the emergence of dual dehalogenaseand luciferase activities.248 The authorscomprehensively studied the dynamics of different variants and usedpartial least-squares regression to identify the strongest predictorsof both activities. This knowledge was then used to obtain a designwith lower product inhibition and highly stable glow-type bioluminescence.In all three case studies above, theauthors used a set of simplesequence-based features such as one-hot encoding of amino acids, physicochemicalproperties of the proteins, or conservation scores. While extractingfeatures from protein dynamics remains challenging see 5.2, a liThese examples are notable because while few groups have the expertiseor infrastructure needed for deep learning, simpler and more accessibleML methods can still be used to advance traditional protein engineeringpipelines. At the same time, we expect that deep learning tools willgradually become more accessible and easier to use.5Despite the excitingapplications and promising case studies discussedabove, several significant knowledge gaps remain to be addressed totake protein engineering to the next level. First, many protein engineeringtasks have yet to benefit from ML, including predicting the effectsof indels and unnatural amino acids, creating predictors that addressseveral targets simultaneously, and predicting mutational effectson protein interactions. Second, molecular dynamics remains isolatedfrom major advances in the application of ML to protein engineering;dynamical information is rarely if ever used when training currentstate-of-the-art predictors. Third, there is a pressing need to establishgold standard protein data sets because such benchmarks have significantlyaccelerated the progress of ML in other domains. Finally, the impactof ML-based tools often remains limited to a narrow circle of methoddevelopers, so there is a need to reach out to the broader communityof biochemists and synthetic biologists. Below we discuss each ofthose gaps in more detail.5.1249 However, current ML-based predictors typicallytarget only one property at a time. This limitation could potentiallybe overcome by combining predictions from multiple tools to definea so-called Pareto front, i.e., a set of solutionsin which no one member is better than another with respect to allobjectives.251 However, by combining the separatelytrained predictors, one misses the opportunity to train on a largerpool of data sets and potentially capture the common underlying mechanismsin a unified protocol. Approaches of this type are rarely used, possiblybecause their implementation would require expert knowledge of multipletypes of data and the experimental techniques used to generate them.One important objective for the future will be to createsingle MLtools that can perform multiple tasks. Multiobjective protein engineeringis a core goal of many ongoing experimental studies because the introductionof mutations targeting one property often affects others. For example,stability and solubility are often negatively correlated and may alsoneed to be traded off with other properties such as catalytic activity.252 Indel engineering isgaining momentum, however, and experimental studies focusing on indelmutagenesis are starting to produce interesting labeled data sets.254 Protein evolutionary information is another potential source ofdata on indels255 that can be used to explorepreviously hidden catalytic activities that could be shifted or promoted.257 Indels can affect not only the structure of the protein system butalso its dynamics.258 Therefore, moleculardynamics simulations may provide valuable data for clarifying theeffects of indels on protein dynamics and their functional consequences.Another major goal is predicting the effects of insertions anddeletions in the amino acid sequence (indels). This area has beenlargely unexplored by ML even though indels occur frequently in natureand can unlock unique functional changes that substitutions alonecannot achieve.259 including multiple point mutants.260 Thesesubstitutions can be used for diverse purposes, ranging from tailoringthe structural, physical, and dynamic properties of specific sitesto introducing new properties by adding carefully designed UAAs. Suchadditions can be used to enhance enzyme activity or elucidate theenzyme reaction mechanisms.261 While thereare some emerging ML-based tools for rational design of UAA sites,262 further research and development are neededto make this approach widely applicable in enzyme engineering.The vastness of the combinatorial protein sequence space can befurther extended by introducing unnatural amino acid (UAA) substitutions.Experimental studies have successfully incorporated over 150 differentUAA substitutions into protein sequences,264 reliable and practically usefulapproaches for designing noncovalent interactions are still missing.For example, Geng et al. found that the evolution of ΔΔGpredictors for protein–protein interactions had been hinderedby the absence of centralized benchmarking.265 Little progress has been made in this area, as evidenced by there-emergence of similar ML models and the persisting lack of commonevaluation standards.267 Furthermore, the reliability of existing models is undermined bya frequent reliance on supervised ML with limited annotated data.268Another area that would benefit from the greater use of ML is thedesign of protein–protein and protein–ligand complexes.Despite recent promising results in the use of ML to predict the bindingsites and protein–ligand complexes,270 or can evenrelate to particular structural features.272 Predictors working on the level of nucleotides or codons may thusbe better tools for protein design and modification, particularlyin areas such as prediction of expression, solubility, and aggregation.The fact that the 64-letter codon alphabet serves to encode richerinformation than the 20-letter amino acid alphabet can be directlyexploited by ML models for improving performance on a wide range oftasks that are now being tackled at the protein sequence level.273 While there have been several studies, e.g., tackling protein expression optimization,274 melting temperatures, subcellular localization,solubility or function,273 we believe furtherresearch in this area might provide a strong boost for predictingmany essential protein characteristics but will require rethinkingof the existing data sets at the nucleotide level.Finally, most ML-based predictors representprimary protein sequencesat the level of amino acids. However, synonymous mutations that donot change the amino acid sequence can still significantly impactprotein expression and function5.2275 The value of analyzing protein ensembles rather than single-proteinstructures has been demonstrated in contexts including predictingthermal stability276 and identifying thelong-range conformational dynamic effect on residues involved in substratereorientation277 and reactivity-promotingregions.278 Therefore, tools that can generatedynamic data without extensive computational or experimental datacollection would be extremely valuable. This has motivated the developmentof new methods for generating representative ensembles from structure279 or sequence data.280Proteindynamics profoundly influence biological phenomena and propertiesranging from enzyme mechanisms to protein stability and must thereforebe taken into account when designing enzymes.e.g., the flexibilityof structural elements expressed in terms of their B-factors, in theset of input features. Another possibility would be to select a subsetof conformations to be used during training. For example, one mightuse apo and holo structures or representative conformations chosenwith various clustering methods or Markov State Modeling. However,the applicability of these methods to training data sets of multipleproteins has yet to be explored.Despite the growing availability of protein dynamics datafrom sources such as MD simulations, current methods for predictingprotein properties are mainly based on single sequences or staticstructures. This is partly because several challenges must be overcomewhen MD results are incorporated into the training and inferencephases of classical ML pipelines. The first challenge concerns datarepresentation: predictive models typically work with static structures.One possible strategy could be to include a limited set of quantitiesderived from a trajectory, 281 which builds on research related to distillation of neural networkmodels.282 In data set distillation, thegoal is to distill a larger data set into a smaller one of potentiallyartificial examples so that models trained on the smaller data setcan match the performance of those trained on the larger one. Thisprocess was first demonstrated in the image recognition domain bydistilling the well-known MNIST data set283 of 60k images into just 10 synthetic images (one per class); modelstrained on the distilled data set achieved performance closely approachingthat of models trained on the original data set.281 At present, data set distillation has been most beneficialfor models that are retrained multiple times, such as those used in continual learning or NN architecture searching. The use ofdata set distillation is conditioned on the existence of large datasets and relevant models that can be trained on those data. So far,relatively little work has been done on distilling temporal data suchas videos,284 but as soon as data set distillationdemonstrates its utility in the domain of video analysis, we expectthis technology to become applicable to processing of MD trajectoriesas well. Data set distillation techniques have recently been comprehensivelyreviewed by Yu et al.285 and Lei et al.286An interesting recent approachthat could help with data processingis “dataset distillation”,287 huge file sizes, and the factthat MD data are rarely published. Furthermore, MD trajectories canbe very sensitive to the choice of force fields288 and other settings. This high variability of simulationdata makes compiling large, consistent, high-quality data sets trulychallenging. The application of the FAIR principles289 to the publication of biomolecular simulation results isthus an important first step toward building relevant data sets andbenchmarks and will encourage further development of the field bygreatly increasing data availability. The initiative of publishingthe MD trajectories is also called upon by Tiemann et al., who performedan extensive MD data mining exercise and demonstrated the utilityof publicly accessible data.290 The prioritizationof building data sets for specific proteins or protein families appearsto be a reasonable next step, followed by exploring the possibilityof transferring knowledge between different protein families.291The lack of training data and benchmarksappears to be anotherfundamental barrier to the systematic integration of MD data intoML-based protein engineering pipelines. Compiling data sets of MDtrajectories presents both technical and scientific challenges resultingfrom the inconsistent file formats due to the use of different simulationpackages,5.3292 For instance, the remarkable progress in imageclassification can largely be attributed to the existence of well-preparedand maintained benchmarks: an early example was the MNIST collectionof hand-written digits,293 which was later complementedby the PASCAL visual object classification (VOC) challenge data setof real photographs293 and the ImageNetdata set.294 MNIST enabled the first successesof deep learning on hand-written digits,295 while the PASCAL VOC data set and the associated challenge establishedstandard benchmarking practices in computer vision research, includingthe use of a hidden test set. Over the past decade, the benchmarkingof progressively more advanced convolutional neural networks on ImageNethas produced models with superhuman classification performance.296 Here it is important to stress the differencebetween a data set and a benchmark data set: to qualify as a benchmark,a data set must satisfy stringent quality criteria, have well-definedbenchmarking tasks and performance metrics, and have a predefinedsplit into training and test sets.297 Moreover,the performance of all models on the MNIST data set is measured interms of the percentage of wrong classifications. Such universallyaccepted benchmarking criteria enable clear and fair evaluations ofthe practical performance of proposed models, eliminate the need tospend time on data preparation, and introduce an element of competitioninto model development.High-qualitydata benchmarks are major drivers of progress in ML research.e.g., MMseqs2 and Foldseek enable fast deduplication andclustering of sequences298 and monomeric structures.298 However, it is extremely difficult for data sets focusing on thecatalytic activity, specificity, enantioselectivity, or solubility.Large-scale data cleaning also remains challenging for protein–proteininterfaces, which are highly repetitive due to the redundancy of thePDB.300While such benchmarking practices arestandard in traditional ML domains, they remain far from common inprotein engineering for several reasons. First, the complexity ofthe domain presents challenges in collecting data and ensuring theirquality. The quality of ImageNet was guaranteed by manual verificationof each image, which was achieved by creating a special interactiveweb site to which any nonexpert could contribute. The bulk of theannotation of one million images was done using annotation services,such as Amazon Mechanical Turk. However, protein data require muchmore involved manual curation by domain experts, making such approachesunusable. Automated curation may be feasible for some types of data, 156 For example, a widely used data set that wasreported to suffer from improper splitting301 is PDBBind.302 Simple random splits mayresult in data leakage, especially when using highly redundant ornonuniform data sets.304Next,the selection of a suitable ML evaluation metric and datasplit may be challenging because of the inherent interdisciplinarityof protein engineering. The establishment of benchmark componentsrequires a deep understanding of the intricacies of biochemistry aswell as expertise in ML. For example, mutational data sets can besplit at the level of specific substitutions, mutation sites, specificproteins, or protein families while ensuring that no entry in thetest set is repeated in the training data. However, even in a simplesetting using protein sequences, splitting at the protein level becomeschallenging if the data set contains many homologs or protein variants,and many authors use different sequence identity cutoffs for clusteringdata before splitting. Protein structure-based learning introducesadditional complexity into the process of defining splits. Distinctsequences may have very similar tertiary structures, necessitatingthe use of more advanced geometric or graph-based splitting conditions.However, such methods are rarely applied, leading to the use of awide variety of splits that limits model comparability.305 It has been produced and maintained by a community-driveneffort that has been ongoing for almost three decades, enabled thesuccess of AlphaFold2, and is continually being extended to new andmore complex tasks.99 Despite the existenceof several other CASP-inspired benchmarks such as Critical Assessmentof Prediction of Interactions (CAPRI) and Critical Assessment of FunctionAnnotation (CAFA),307 standard benchmarks for proteindesign are still lacking. This problem is attracting attention; forexample, Dallago et al. recently introduced the Fitness LandscapeInference for Proteins (FLIP) benchmark,303 which addresses three protein engineering problems by including:(i) an almost complete mutational landscape (149 361 of 160 000variants) of four strongly epistatic residues of the G protein, (ii)the more diverse and sparsely sampled landscape of the AAV capsidprotein, and (iii) the highly diverse thermostability landscape ofproteins from different domains of life. Similar benchmarks are alsoappearing in related disciplines such as genomics308 and drug discovery.309 Calls for definingclear criteria for the study of epistasis and residue coevolutionpatterns can be found in the literature;175 these criteria could enable the emergence of new benchmarks. Wehope to see more initiatives creating and supporting benchmarkingin the future, as they could dramatically accelerate progress in MLfor protein design.The Critical Assessment of Structure Prediction (CASP) isthe bestexample of a well-established benchmark in protein science.5.4e.g., biochemists. One established scientific publishing standard fornew ML methods is that protocols and scripts should be included withthe submitted manuscript.310 While thisimproves the peer-review process, potential future users of such toolsare generally less comfortable running such scripts than reviewers.Therefore, if a new method is available only as a collection of scriptsin a GitHub repository, it is unlikely to have much impact outsidethe narrow community of ML developers.Even when authors have performed appropriateindependent method validation and convinced readers of the superiorityof their tool, the transfer of knowledge from published methodologiesto new applications seems to be frustratingly slow. Despite the rapidlygrowing number of publications describing applications of ML in proteinengineering, surprisingly few studies have followed up on publishedmethods and applied them to new protein targets. The main obstacleto such work appears to be the limited accessibility of ML methodsto researchers without expertise in computer science, 311 allowing it to be used by researchers in a way that ismuch more user-friendly than downloading code from GitHub. In LoschmidtLaboratories, we have been developing user-friendly tools for overtwo decades , and we often hear how crucial the ease of use of these tools isto our users. We ensure this ease of use by making efforts to supportpopular input formats , creating intuitive and well-guided settings , and providing comprehensive and easily understood reportingof results. We also strongly encourage other developers to investtime into making their tools accessible to other communities, suchas enzymologists or biochemists. Moreover, it is usually helpful tocontact past or potential users to understand how they see the methodand what functionality they would benefit from in addition to themethod itself. This could include fetching sequences and structuresfrom databases, submitting multiple sequences or mutations as a singleinput, suggesting designs for experimental validation, or even allowingoutput graphics to be easily transferred to a publication.Creating at least a minimalisticuser-friendly interface is thus vital for making methods widely accessibleto users other than their developers, even though it generally requiresadditional work from the developers that is outside the scope of theirresearch objectives. The explosive growth in the number of users ofChatGPT 3.5 and later versions has been partially attributed to itssimple dialogue-like interface, which is accessible even to userswithout knowledge of ML. Similarly, the AlphaFold2 release was followedby the dissemination of a Google Colab notebook,e.g., manual fine-tuning, these should bediscussed explicitly in the main text of a publication or in instructionsfor using tools. While there is pressure to present a tool’sperformance in the most favorable light, any cherrypicking will eventuallydisappoint future users and undermine the trust in the entire domain.To be successful, ML requires users and the wider community to trustits predictions and be convinced that the tools have been assessedfairly. Consequently, there are active efforts to improve reportingstandards for ML tools in modern biology, and several clear guidelineshave recently appeared to guide authors in preparing manuscripts dealingwith ML.312 However,there will be a lag phase before such rules are universally adopted.Wealso encourage authors to be more open about the limitationsof their tools in publications. When published case studies use additionalsteps, 66.1In many domains, including medicine andfinance, it is vital forML systems to be interpretable and explainable to build trust in thealgorithms. Understanding the mechanisms behind making a predictioncan also enable more rigorous verification of a tool or provide cluesfor follow-up decision-making. In this context, an explainable modelis one that provides insights into the reasons for its predictionsand decisions, for example, by showing which parts of the input havethe greatest impact on predictions. Conversely, an interpretable modelis one for which the internal process of making predictions can bereadily understood by humans. This can be achieved by having an explicitdecision pathway in a decision tree or easy-to-grasp feature weightsin a simple linear equation. If ML algorithms are not mathematicallycomplex, they are often intrinsically explicable and interpretable.However, interpreting and explaining their predictions becomes morechallenging as the algorithms become more complicated. This is whymodels such as deep neural networks are often described as black boxes.The field of explainable artificial intelligence (XAI) seeks to overcomethis challenge by creating explanations using analytics, saliencymaps, or words to allow humans to understand why an ML algorithm makesa certain decision or prediction.314  = −ΔΔG(Mut → WT).That is, the change in ΔG upon mutating a residueis equal to the negative change in ΔG thatwould be induced by the hypothetical inverse mutation. This propertymakes it possible to augment the training data set by adding artificialinverse mutations from the mutant sequence (Mut) to the wild-typesequence (WT) to obtain a balanced data set. Antisymmetry can alsobe incorporated by design, ensuring that the property is enforcedeven when using an imbalanced training set.339 Interestingly, efforts to solve the problem of predicting proteinstability changes upon mutation have driven progress in other sensitivitystudies, e.g., the use of reduced amino acid alphabetsto account for bias in the representation of mutations in the trainingdata,131 structural sensitivity,340 or the “scaffold bias” of usingcrystallographic structures instead of AlphaFold models in ensembles.276 Another task where data set biases were reportedis structure-based virtual screening. This problem was recently tackledusing an ML-based scoring function in which the authors took careto ensure that feature importance was consistent with human knowledge;this forced the model to learn relevant features regardless of thepresent biases, leading to better generalizability.341A typical example of a bias in enzymaticdata arises in the task of predicting mutational stability, wherea predictor given an original and mutated sequence as inputs shouldreport a single number corresponding to the predicted ΔΔG.In general, a random mutation of a natural protein is more likelyto be destabilizing than stabilizing,342 The aim of multicalibration is to make predictorsperform more uniformly across different data subclasses. However,the complexity of the proposed approach is linear with respect tothe number of possible subsets of the training data and, thus, exponentialwith respect to the number of training samples. To alleviate thishigh computational cost, other studies proposed low-degree multicalibrationby drawing on the multicalibration and multi-accuracy approaches.344 These concepts are currently mainly being studied within the domainof algorithmic fairness,345 which is largelyconcerned with ML-based predictors that process personal data on humanindividuals. It is interesting to see if these concepts will be usefulin the context of enzymology, for example, by ensuring that learnedpredictors have similar performance for different protein families.In many cases, however, even a detailed understandingof the studiedtask will not reveal straightforward paths for uncovering and fixingbiases because problems often arise from intricate interdependenciesbetween data points that researchers are unaware of. To combat suchissues, the concept of multicalibration has been proposed.6.338 areshining examples of this trend. To increase the size of the data setsavailable for model training in protein engineering, it may be necessaryto use as much available experimental data as possible, acceptingthat experimental data sets will inevitably vary widely in qualityand size. To bridge the gap between small high-quality and large low-qualitydata sets, meta-learning appears to be a promising strategy. In suchan approach, nested optimization could be employed for training ona large set of noisy examples in the inner loop and a small set oftrusted examples in the outer loop, thus suppressing the impact ofthe noise in the larger data set.346One currenttrend in machine learning is that models are increasing in size inparallel with the increasing amount of data on which they are trained.Models such as ChatGPT and DALL-E347 The hardware requirementsof MD using such large data sets could be met using purpose-builtsupercomputers such as Anton 3.348 Advancesin quantum computing (QC) technology could also make large-scale molecularsimulations more accessible, as well as benefit generative ML tasks349 and improve the generalizability of modelstrained on limited data.350 However, thegreatest benefit of QC in enzyme engineering will likely result fromits expected ability to greatly accelerate quantum and molecular mechanicssimulations.352Molecular simulations using MD and quantum mechanical methods couldbe another valuable source of data for training large, data-hungrymodels. However, these simulations are very computationally demanding,as demonstrated by Musaelian et al., who recently used machine-learnedpotentials in simulations of biomolecules using 5120 A100 GPUs inparallel.2 emissions resultingfrom the training and inference process.353 This issue can be addressed by adopting energy-saving ML practices,354 implementing architectural redundancy to bypasslengthy training processes,355 or obviatingthe need for large model training by narrowing problems down to simplelinear combinations of system-relevant physical properties.356 Another important recent trend in machine learninginvolves lightweight adaptation of large pretrained models to newtasks using adaptor layers.357 These methods changeonly a small fraction of the parameters of the large scale pretrainedmodel for the new task, leaving the vast majority untouched. Thisallows large models to be adapted to new tasks at a fraction of theirinitial training costs; the time required for adaptation may be aslittle as several hours on a single eight-GPU machine,359 eliminating the need for the supercomputer infrastructure used totrain the original model. This significantly accelerates model adaptationand makes such training and related research accessible to a widecommunity of researchers who may lack supercomputer access.Given the trend towardever-larger models, it is important to considerthe cost of their training and the COreinforcementlearning (RL) to select structural or biophysical motifs thatare important for a target property.360 Several recent advances in RL have focused on small molecule design.361 However, when combined with recent advancesin protein science, RL has also shown promise in the design of proteincomplexes362 or peptide binders with highaffinities.363 Based on this expansionof its applicability domains, we believe that RL could be similarlyuseful in enzyme engineering. Enabling more user-friendly pipelineinteraction, for example, by creating ChatGPT-like interfaces to simplifythe control and running of experiments, can potentially further contributeto the automation of the workflows.Another promising area of application for ML is further automationof enzyme engineering pipelines, for example, by using 7We live in exciting timesthat provide many opportunities to explorenew horizons in enzyme engineering and machine learning. Many ambitioustasks are already being tackled by cutting-edge data-driven algorithmsand approaches, often inspired by their spectacular performance inother domains. The tools created on their basis are already playingintegral roles in studies aiming to discover and improve biocatalysts.On the other hand, many more goals are still waiting for the appearanceof suitable data sets and data processing protocols to be eventuallyleveraged by machine learning. There are many challenges ahead, includingthe creation of reliable and user-friendly tools for generating promisingprotein designs by users with limited ML knowledge, addressing multipletasks in one pipeline, incorporating protein dynamics in ML pipelines,understanding the effects of insertions, deletions, or unnatural aminoacids, increasing the interpretability of the models, and revealinghidden data biases. One of the most significant takeaways from thesuccess of machine learning in other domains, such as natural languageprocessing or computer vision, is the role of large-scale data collectionand curation. As biochemistry data tend to be more challenging toacquire than, for example, images or text, developing mechanisms foropen sharing and aggregation of data sets across the entire scientificcommunity could have game-changing effects. Building and maintainingcommunity-wide training data sets and benchmarks, including such ambitiousdata types as protein dynamics, could create entirely new ways fordesigning proteins that would shape the future of biotechnology.