A survey of machine learning techniques in structural and multidisciplinary optimization

Abstract

Machine Learning (ML) techniques have been used in an extensive range of applications in the field of structural and multidisciplinary optimization over the last few years. This paper presents a survey of this wide but disjointed literature on ML techniques in the structural and multidisciplinary optimization field. First, we discuss the challenges associated with conventional optimization and how Machine learning can address them. Then, we review the literature in the context of how ML can accelerate design synthesis and optimization. Some real-life engineering applications in structural design, material design, fluid mechanics, aerodynamics, heat transfer, and multidisciplinary design are summarized, and a brief list of widely used open-source codes as well as commercial packages are provided. Finally, the survey culminates with some concluding remarks and future research suggestions. For the sake of completeness, categories of ML problems, algorithms, and paradigms are presented in the Appendix.

Appendix. ML methods widely used in the context of structural and multidisciplinary optimization

ML algorithms can be categorized into four groups: 1) classification, 2) regression, 3) clustering, and 4) dimension reduction as shown in Fig. 6. Classification and regression are both supervised learning algorithms, where the main idea is to generate a prediction model. If the predicted response is discrete, it is a classification problem, whereas if the response is continuous, then it is a regression problem. Therefore, in general, the ML algorithms used for classification and regression are very similar. The most commonly used classical ML algorithms for classification problems include logistic regression [Cox (1958)], k-nearest neighbors [Fix and Hodges (1989)], support vector machines (SVM) [Cortes and Vapnik (1995)], kernel SVM, naive Bayes, decision tree classification, and random forest classification. The most commonly used classical ML algorithms for regression problems include simple linear regression, multiple linear regression, polynomial regression, Kriging, support vector regression (SVR), decision tree regression, and random forest regression.

Fig. 6

Categories of ML problems

Clustering is similar to classification in that they are both used for grouping the data. The main difference is that classification is used to categorize labeled data, whereas clustering detects patterns within an unlabeled data set. Therefore, the classification is a supervised learning algorithm, whereas the clustering is an unsupervised learning algorithm. The most commonly used classical ML algorithms for clustering problems include k-means, mean shift clustering, Gaussian mixture models, density-based spatial clustering, and hierarchical agglomerative clustering.

Dimension reduction aims to reduce the number of input variables in a dataset, thereby protecting against the curse of dimensionality, which makes the algorithm difficult to run as the dimensions of the data increase. Data from a large dimensional space is transformed into a smaller dimensional space ensuring that it provides similar information. Dimension reduction methods can be further categorized into linear methods and non-linear methods. The most commonly used linear learning algorithms for dimension reduction include principal component analysis [Wiener (1938)], factor analysis [Harman (1976)], linear discriminant analysis [Fisher (1936)], and singular value decomposition [Golub and Reinsch (1971)]. The nonlinear algorithms include kernel principal component analysis, isometric mapping, and t-distributed stochastic neighbor embedding (t-SNE). Among the ML methods listed in Fig. 6, we briefly explain ML methods that are widely used in the context of structural and multidisciplinary optimization in the following subsections.

1.1 A.1 Linear regression

Linear regression [Montgomery et al. (2021)] models the relationship between the response variable (dependent) y and one or more independent variables x. If there exists only one independent variable, then it is called simple linear regression. The fundamental idea in linear regression is to find the coefficients of the basis functions that best model the data. Ordinary least squares (OLS) are the most common method used to train the model with the given data to estimate the unknown coefficients. Function nonlinearity is modeled using complex basis functions while keeping the regression linear.

$$y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \cdots + \beta_n x_n + \varepsilon = {{x}}^{{f T}} {\boldsymbol{\upbeta }} + \varepsilon$$

(A1)

where β₀ and β_n are the unknown coefficients and ε is the error term.

1.2 A.2 Gaussian process (GP)

GP [Rasmussen (2003)], also known as Kriging when the mean of GP is zero, is a stochastic approach that finds wide use in regression, classification, and unsupervised learning. It is usually utilized in the linear regression framework while using the Gaussian kernel as the basis function. It is the preferred approach for inference on functions as well. GP is a generalization of Gaussian probability distribution in which every finite collection of random variables has a multivariate Gaussian distribution. GP is a distribution over functions with a continuous domain such as time or space. Since GP provides model prediction as well as prediction error estimates, even when the simulation is deterministic, it is sought after to be used as surrogates in design and analysis of expensive computer experiments. Since GP metamodels can fit complicated surfaces well, it is suited for fitting accurate global metamodels. A GP is completely specified by mean function m(x) and covariance function $k\left( {{{\bf{x}}},{{\bf{x}}}^{\prime}} \right)$ as

$$f\left( {{\bf{x}}} \right) \sim GP\left( {m\left( {{\bf{x}}} \right),\,k\left( {{{\bf{x}}},{{\bf{x}}}^{\prime}} \right)} \right)$$

(A2)

GP can be extended to multiple outputs by using multiple means and covariances. It permits easy interpolation of data and has an inbuilt mechanism to account for noise. Furthermore, GP can quantify the uncertainty about the prediction and have conditional distributions that allow adaptive sampling or Bayesian studies. Owing to the fact that GP models are regularly evaluated on a grid leading to multivariate normal distributions and the computational time for calculating the inversion and determinant of n × n covariance matrix is of O³, using GP is a challenge while using large scale datasets. Recently, approaches such as matrix–vector multiplication [Gardner et al. (2018a), (2018b)], [Dong et al. (2017)] and sparse GP [Cutajar et al. (2016)] have been developed to reduce the amount of computation when the data set is more than 100 k.

1.3 A.3 Artificial neural network (ANN)

In the 1940s, [McCulloch and Pitts (1943)] formulated the first NN model. Since its inception, NN has found interest among both researchers and applications in various domains. As a result, better algorithms and more powerful networks have been developed. ANN refers to a biologically inspired sub-domain of artificial intelligence (AI) modeled based on the network of the brain. Akin to the human brain, ANNs have neurons (called nodes) which are connected to each other in different layers of the networks as shown in Fig. 7. The basic idea of ANN is that an input vector x is weighted by w and along with bias b, subjected to an activation function f that is linear or nonlinear to produce the output y as given as

$$y = f\left( {{{\bf w}}^{{\rm T}} {{\bf x}} + {{\bf b}}} \right)$$

(A3)

The weights in Eq. (A3) are optimized during training until a specified level of accuracy is reached by the network. Based on the application, there are many activation functions used in ANN, namely sigmoid, hyperbolic tangent, rectifier linear unit (ReLU), Heaviside, signum, and softmax functions [Karlik and Olgac (2011)]. Researchers have also developed application specific activation functions (Wuraola and Patel 2018, [Gomes and Ludermir (2013). Since ANN deals with multidimensional data, approaches such as StandardScaler, RobustScaler, MinMaxScaler, and Normalizer for data scaling, can be used for data processing and can prevent convergence to zero or diverge to infinity during the learning process.

Fig. 7

Simple architecture of ANN

ANN is broadly classified into two categories such as feed-forward NN and feed backward NN. In the feed-forward NN, the information will pass only in the forward direction i.e., from the input layer to the hidden layer (if any) and then to the output. Single-layer perceptron, multi-layer perceptron, and radial basis function networks are examples of feed-forward NN. In the feed backward NN, the inputs are fed in the forward direction and errors are computed to be propagated in the reverse (hence the terminology back) direction to the previous layers, so as to reduce the error in the cost function by readjusting the weights. Examples include Bayesian regularised NN and Kohonen’s self-organizing map. The loss function is computed as the difference between the prediction and the target after each feedforward pass. In the backpropagation process, the optimizer trains parameters such as weights and biases iteratively through optimization to minimize the loss function.

ANNs can be used for both regression and classification problems which are techniques in predictive modeling. In the context of classification, since ANN works by splitting the problem into layered networks of simpler elements, ANNs are reliable when the tasks involve many features. The most attractive feature of ANNs is that they provide predictive capability by mapping any number of inputs and outputs. Upon training, the predictions are fast and cheap.

NNs are typically black box approaches. That is, one might not be able to capture the influence of independent variables on dependent variables. Overfitting is a fundamental challenge of ANN as it depends predominantly on training data. With traditional CPUs, ANNs were expensive in terms of computational time to train the network, but the invention of cloud computing and increased computing power have relieved the computational burden. However, researchers began to focus on more complex problems and used more layers to train on large sets of data, resulting in longer computational times with multiple training iterations.

1.4 A.4 Deep neural network (DNN)

DNN is created when NNs are stacked one after the other. The primary difference between the conventional NN and DNN is that the former has one or two hidden layers and the latter has several hidden layers as shown in Fig. 8. Each circle in the figure calculates a weighted sum of the input vectors and bias following which a nonlinear function is applied to obtain the output. DNNs can handle functions with limited regularity and are powerful for high-dimension problems. The basic idea of DNN is to approximate a function with a non-linear activation function [Emmert-Streib et al. (2020)], with n hidden layers as represented in

$${{\bf DNN}}\left( y \right) = {{\bf w}}^{\left( n \right)} {{\bf x}}^{\left( n \right)} + {{\bf b}}^{\left( n \right)}$$

(A4)

and

$${{\bf x}}^{\left( {k + 1} \right)} = \sigma \left( {{{\bf w}}^{\left( k \right)} {{\bf x}}^{\left( k \right)} + {{\bf b}}^{\left( k \right)} } \right),\,\,\,\,\,\,k = 0,\,1, \cdots ,n - 1$$

(A5)

where w and b are the weights and biases of the network and $\sigma$ is the activation function. DNN is more complex in connecting layers than a network with 1 or 2 hidden layers and has the automatic feature extraction capability. Therefore, when larger training data is used, the DNN can provide accurate predictions compared to classical ML algorithms where the accuracy is kept fairly constant.

Fig. 8

Architecture of DNN

There are three major classes of DNNs, namely supervised, semi-supervised, and unsupervised DNNs. Examples of supervised learning algorithms include deep feed-forward networks (DFNNs) and CNNs. Restricted Boltzmann machines, autoencoders (AEs), GANs, and long short-term memory networks (LSTMs) are examples of unsupervised learning algorithms. Recurrent neural networks (RNN) are an example of semi-supervised learning techniques.

While solving complex problems such as image classification, natural language processing, and speech recognition, DNN is more useful than shallow networks. DNNs typically outperform other approaches when the data is large. DNN architectures are very flexible to adapt to new problems and can work with any data type. Getting trapped in the local minima, vanishing gradient, and overfitting are some of the challenges associated with DNN training that require large data.

1.5 A.5 Convolutional neural network (CNN)

One of the most widely used DNNs are the CNNs [Fukushima (1988)]. While ANN is inspired by the human brain, CNNs are inspired by the human optical system and are predominantly applied to imaging analysis. CNNs consist of two operations, namely convolution and pooling. Unlike ANNs, in CNNs the neurons in one layer are connected to nearby neurons in the next layer. This leads to a significant reduction in the number of parameters in the network. A typical CNN consists of an input, an output, and multiple hidden layers which consist of a series of convolutional layers (filters or convolution kernels) as shown in Fig. 9. ReLU is the typical activation function used, followed by operations such as pooling layers, fully connected layers, and normalization layers. Backpropagation is used for error minimization and weight adjustment. [Wu (2017)] provides a tutorial on CNN. Compared to CPU-based architectures, CNNs with GPU-based architectures take less time for training, because GPU vastly is superior in the computation of dense algebraic kernels, such as matrix–vector multiplication, in which DL algorithms are mainly composed.

Fig. 9

Architecture of CNN

CNNs can easily process high-dimensional inputs such as images. CNNs are good at extracting local information from the text and exploring meaningful semantic and syntactic meanings between phrases and words. Also, the natural composition of text data can be easily handled by a CNN’s architecture. CNNs need large data for training and hence are computationally intensive. Encoding the position and orientation of objects is still a challenge in CNN.

1.6 A.6 Reinforcement learning (RL)

RL [Sutton and Barto (2018)] is one of the paradigms of ML algorithms where the agents learn by interacting with the environment. RL works on trial and error-based learning and maximizes the reward rather than finding the hidden structure unlike other ML algorithms. As can be seen in Fig. 10, an RL agent performs an action ‘a’ while transiting from a state s_t to s_{t +1} and is rewarded r_{t +1} for the action a_t and this process is repeated iteratively to maximize the reward. The probability of transition to the new state is expressed by P(s_{t +1} | s_t, a_t). The best sequence of actions that an RL agent can make is called a policy and the entire set of actions from start to finishthat an agent performs is called an episode. Usually, the dynamics of the RL problem can be captured by using a Markov decision process.

RL usually performs better in solving complex problems compared to other standard learning techniques. If no training data set is available, it is bound to learn from experience. RL focuses on achieving long-term results that are difficult to accomplish by other techniques. Similar to other ML techniques, RL requires large data and is computationally expensive. RL can be heavily affected due to the curse of dimensionality. When the conventional RL is combined with DL, deep RL can be set up. The deep RL uses DNNs to calculate rewards, and policies that are usually accomplished by a state of action pairs in RL. The deep RL can be employed where there exists a complex state and very high computations are required (Fig. 10).

Fig. 10

Basic setting of RL

1.7 A.7 Recurrent neural network (RNN)

RNN [Rumelhart et al 1986] is one of the common semi-supervised learning algorithms that use sequential data (or ordered data) for training. Examples of ordered data are DNA sequence, financial data, and time-series data. RNN uses the current input as well as the past history of inputs that it has learned through the hidden state while making decisions. Typically RNNs consists of an input layer, a hidden layer, and an output layer as shown in Fig. 11. The number of neurons in the hidden layer of RNNs should be between the number of inputs and the number of outputs. The key feature of RNN is that it makes a copy of the output and sends it back into the network. Thus, the past information gets stored. The change in the knowledge of the network is updated in the hidden state at every time step and the update can be expressed as

$$h_t = f_w \left( {x_t ,\,h_{t - 1} } \right)$$

(A6)

where h_t is the new hidden state, h_t-1 is the past hidden state, x_t is the current input, and f_w is the fixed function with trainable weights.

Fig. 11

RNN with hidden memory state

These algorithms commonly find application in ordinal or temporal problems such as image captioning, speech recognition, and natural language processing. Similar to spatial data being efficiently processed by CNNs, RNNs are designed to process the sequential data in an efficient manner. As the number of time steps increases, the number of model parameters in the RNN model does not increase. While training an RNN, error gradients are used to update the network weights. Sometimes the error gradients can accumulate resulting in large updates of weights (exploding gradients) and an unstable network. On the other hand, if the weight updates are small, one faces the problem of vanishing gradients. These are the two major issues associated with the RNNs. In order to solve the gradient problem, weight initialization methods such as Xavier initialization and He initialization, gradient clipping, and batch normalization are used, or an LSTM or GRU is devised. When timely dependencies in sequences need to be captured, RNN are one of the best choices. However, recent developments such as Transformers [Vaswani et al, (2017)] can outperform RNN in such applications.

1.8 A.8 Variational autoencoder (VAE)

An autoencoder (AE) is a type of unsupervised learning that learns unlabeled data and has traditionally been used for dimensionality reduction and feature learning, but recently it has gained a lot of popularity as a generative model that can generate data similar to training data. Since VAE (Kingma and Welling 2013) is based on an AE, it consists of two parts: encoder and decoder. However, unlike AE, which represents a latent vector as a value, the latent vector of VAE uses a density function. Since VAE is based on a probabilistic model, it has computational flexibility. This latent vector is used to predict an input image, and VAE training is performed with the goal of reducing the difference between the generated image and the input image as shown in Fig. 12. Finally, evidence lower bound and re-parameterization tricks are used to perform optimization. The main advantage of VAE is that it is useful to perform other tasks such as design optimization in the latent space using the latent vector information. However, since the density is not obtained directly, the quality of the generated model may be somewhat inferior to the direct density methods such as pixelRNN or pixelCNN, and the generated image is relatively blurry compared to GAN.

Fig. 12

Architecture of VAE (Asperti et al. 2021)

1.9 A.9 Generative adversarial network (GAN)

GAN [Goodfellow et al. (2014)] as shown in Fig. 13 trains a model that samples a latent vector from a simple distribution and generates it as an image based on the game-theoretic approach. The objective function of GAN consists of a discriminator output for real data, and a discriminator output for generated fake data. Because the generator and discriminator train with the goal of minimizing and maximizing the objective function, respectively, GAN is called a minmax game. The story of a counterfeiter (generator) and a police officer (discriminator) is an easy-to-understand example of the concept of GAN. When a counterfeiter creates a counterfeit currency, the police can determine whether it is genuine or not, and in the process, the generator and discriminator evolve competitively to generate a more authentic counterfeit currency.

Fig. 13

Architecture of GAN (Alom et al. 2019)

GAN is difficult to apply to various fields due to unstable learning ability; consequently, a DCGAN [Radford et al. (2015)] with CNN in the generator part was developed. In addition to the GAN model, various models such as conditional GAN (cGAN), boundary equilibrium GAN, and super-resolution GAN have been developed to improve performance and for application to new fields. The main advantage of GANs is that it is possible to create new and novel images.

1.10 A.10 Ensemble methods

Ensemble methods are one paradigm of ML techniques that have become popular during the past three decades [Bishop (1995)], where several learning algorithms are used to train and solve a problem. Boosting, bagging [Bühlmann (2012)], and stacking (Džeroski and Ženko 2004) are the most widely used approaches in ensemble methods. AdaBoost [Rätsch et al. (2001)], gradient boosting (Friedman 2001), extreme gradient boosting [Chen and Guestrin (2016)], and light gradient boosting [Ke et al. (2017)] are a few algorithms that are more frequently used in boosting. Bagging meta-estimator and random forest are the popular ensemble algorithms in bagging.

Ensemble methods are used to improve the accuracy of the model by reducing the variance. Bagging is a variance reduction technique whereas boosting and stacking’s objective is to reduce the bias and not the variance. Ensembles have been shown to serve as insurance against bad predictions and issue a red flag when one of the models is performing inconsistently on a consistent basis, especially at regions of interest. Eventually, all the ensemble algorithms attempt to improve the model accuracy. The generalization ability of a single learner is not as good as ensemble methods, since it uses multiple learners, and this is one of the major advantages of using ensemble methods.