Performance comparison of five regression-based machine learning techniques for estimating load-carrying capacity of steel frame using direct analysis

. The development of computer science has promoted the application of scientific and technological achievements to construction engineering in general and steel structure design in particular. Recently, the steel frame structure design has applied advanced analytical methods to take into account the inelastic behavior of the material and the geometric nonlinear properties of the structure, leading to the results obtained close to the real structure. In addition, to reduce computational time and effort, it has been applied advanced machine learning (ML) algorithms to predict behavior, helping to accelerate decision-making, improve efficiency, and reduce errors. In this paper, 5 popular machine learning algorithms are currently being conducted for the regression-based to estimate the ultimate load capacity of steel frames, including Linear Regression, Deep Learning, Support Vector Machine, Random Forest, and XGBoost. The effectiveness of applying these methods is tested through a numerical example surveying a 20-story space steel frame. The performance of ML algorithms is evaluated by comparing the mean-squared error (MSE) and computational efforts. The results show that among the 5 selected methods, the XGBoost has obvious advantages and is superior in terms of both MSE and calculation time.


Introduction
Steel frames are commonly used in both civil and industrial buildings because they can meet the multi-objective in construction such as performing large spans and spaces, and diverse forms of different types of structures and buildings.Mention in the field of designing steel frames, it is necessary to consider the influence of geometrical nonlinearity and/or inelasticity of the material, as well as the failure modes of the structural system.To achieve this purpose, non-linear inelastic analyses are applied to directly evaluate the response of the whole structural system and thereby evaluate the ultimate load-bearing capacity of the structure.However, the above-advanced analysis is more time-consuming than the traditional elastic analysis.This even more challenging to deal with complex problems such as optimization design, structural reliability analysis, and damage assessment.These algorithms must perform a large number of iterative analyses to converge to the final solution.
In recent years, machine learning (ML) methods, through low-cost predictive models and decision support, are now an alternative to problem-solving.Different from conventional techniques, ML offers several advantages, especially when dealing with very complex problems, such as speeding up the decision process, reducing error rates, and increasing efficiency.Using the ML technique, a meta-model is built through training data which is created by using structural nonlinear inelastic analysis.This meta-model is an approximate mathematical representation of an approximation mathematical function used to describe a high degree of abstraction and complexity in data.In the meta-modal, the structural nonlinear responses can be approximately estimated without the need for recalls time-consumed nonlinear inelastic analysis.Therefore, the time and computational resources of design problems that require many times of structural analyses will be significantly reduced.A review of a large number of the applications of ML in structural engineering in [1,2].
Currently, the implementation of ML algorithms to predict the load-bearing capacity of structures has attracted more and more attention from researchers.For example, the loadbearing capacity of concrete-filled steel tube columns has been estimated using 4 popular ML algorithms [3].The load capacity of the reinforced concrete rib-slab system is predicted using Extreme Gradient Boosting (XGBoost), which provides a better description than existing techniques outlined in Euro Code design standards [4].While ML algorithms have been well applied to evaluate the structural performance and maximum load capacity of different building types as a bridge [5], trusses [6], beams [7], and columns [8] …, there is still limited research on steel frames exhibiting higher order inelastic responses.
In this study, 5 well-known ML algorithms including linear regression (LR), Support Vector Machine (SVM), deep learning(DL), random forest(RF), and extreme Gradient Boost (XGBoost) to be used to estimate the ultimate load -bearing capacity of steel frame with medium computational effort.The space steel frame of 20 stories was surveyed to review and compare the effectiveness of the methods.The data set is generated by performing nonlinear inelastic analyses.In the dataset, the input is the geometrical features of the beam, and column sections are W_shaped and be chosen from a list.One output is the ultimate load factor (ULF) of the structure.The amount of learning data varied from 1,000 to 20,000 in all the study algorithms.The conclusion illustrates and summarizes the performance of the selected ML methods.

Review of 5 machine learning algorithms
Regression analysis is used in this research to predict the load-bearing capacity of steel frame through Ultimate load factor (ULF) is a supervised regression problem because the input is the available cross-section and the output is known.Various regression techniques in machine learning such as linear, SVM, RF, ANN, DL, and Tree Boosting …are available.A brief overview is presented in this section for 5 popular regression techniques.The detail of each method can be seen in the references related.

Linear Regression (LR)
Linear regression (or least squares fitting) is the simplest and most commonly applied when dealing with data in engineering [9].The relationship between input data and output data can be represented by a linear function.Solving a linear regression problem is essentially the process of finding a set of weight parameters so that the error between the true value and the predicted value is minimal.Although it is possible to find an analytical solution, in practice for large amounts of data, the above problem is often solved by the technique of Gradient Descent.With the training data set, the most common method for estimating the weight set is to minimize the residual sum of squares (RSS).The least squares estimates of the parameter β have the smallest variance among all the linear unbiased estimates.

Support vector machine (SVM)
Support Vector Machine (SVM) is a supervised learning machine learning algorithm that can be used for both classification and regression challenges.SVM is mostly used in classification problems.In the SVM classifier, each data item is plotted as a point in ndimensional space (where n is the number of features), with the value of each feature being the value of a particular coordinate.Then, performed classification by finding the optimal hyper-plane to segregate into classes.The SVM in regression is to find a function that approximates mapping from an input domain to real numbers based on a training sample [9].

Deep Learning (DL)
DL algorithms or deep neural networks (DNN) constitute a branch of machine learning algorithms and are very efficient in handling both regression and classification problems for structural engineering applications.Due to the features of training data, the learning form of a DNN model can be classified as (1) supervised learning, (2) unsupervised learning, and (3) semi-supervised learning.In the current study, only supervised learning is taken into consideration since the input and output data are specified.
A feedforward neural network (FNN) includes many layers and one layer has many neural.The first layer plays the role of the input layer in which each neural is a data feature.In the input layer, the number of neural and the number of data features can be equal or different from each other.The last layer is the output layer where each neural is a data label.The number of neural and the number of the data label must be identical.The remaining layers are hidden layers, each of which contains many neural.The role of the hidden layers is to transfer the important features from the input layer to the output layer.In the FNN, the information only transfers in one direction from the input layer, through the hidden layers, to the output layer as presented.In multilayer FNN, a BP algorithm is commonly used to compute the influence of each weight corresponding to a loss function based on the gradient descent method.In the BP algorithm, the backward pass is implemented to update the weight functions from the output layer through the hidden layers to the input layer to search for the new weights so that they minimize the loss function [10].

Random forest (RF)
Random Forest (RF), proposed by Breiman [11], is a group of un-pruned regression or classification trees made from the random selection of the training data samples.In the induction process, random features are to be chosen.By aggregating (majority vote for classification or averaging for regression) the predictions of the ensemble, the prediction is made.A pictorial representation of RF is described in Fig. 1 [12].The RF is based on the methods of Bagging, Randomizing Outputs, and Random subspace excusing boosting.

Extreme Gradient Boosting (XGBoost)
Extreme Gradient Boosting (XGBoost) is boosting algorithm which is an improvement on Gradient Boosting Decision Tree (GBDT).Strong classifiers with good performance are obtained by a series of weak classifiers.AdaBoost, the abbreviation of "adaptive boosting", was proposed by Freund and Schapire in 1995 [13].In the initial training step, the weights of all samples are the same, and then the weight of the samples which are wrongly classified by the weak classifier will be strengthened.Because the weight is increased in the error function, the next weak classifier will tend to classify the weighted ones correctly to reduce the classification error.Different from AdaBoost, every calculation of GBDT is to reduce the last residual and then establish a new model in the direction of residual reduction (negative gradient).Chen and Guestin developed XGBoost in 2016 [14] based on GBDT focusing on computation speed and model performance.In XGBoost, the regularization technique is applied to avoid overfitting and harmonizing the parameters.For example, it allows a feature sampling method to prevent overfitting.At the same time, it also improves the loss function and calculation efficiency.
3 Estimating load-bearing capacity of steel frame using advanced analysis

Load -bearing Capacity of steel frame
The load-bearing capacity of a steel frame can be represented by using the ULF which is defined as the ratio of structural resistance (R) to loading effect (S).Based on the value of the ULF, the safety of the frame is assessed.If the ULF is smaller than 1.0, the structure is considered to fail; otherwise, the structure is safe.
In this study, ULF was calculated by non-linear inelastic analysis.This method can directly determine the structural resistance R in terms of ultimate load capacity without requiring the entire response spectrum of the structure.

Advance Analysis of steel frames to estimate the ULF
In the context of modeling the steel frame structure which accounts for nonlinear inelastic analysis for ULF estimation, the beam-column method is used in this research.This approach requires lower computational effort but introduces a certain amount of error since the plastic response is localized in certain areas.Nevertheless, it is used in many cases for structural design.In this approach, each structural element beam or column is simulated as a plastic-hinge beam-column element, at the ends of which plastic hinges can develop to account for the nonlinear response.Apart from this, stability functions are used to capture second-order effects [15,16].For the members under an axial force, the Column Research Council (CRC) tangent modulus concept [17] is applied to estimate the effects of residual stresses.For the members under axial force and bending moments, the gradual stiffness degradation model [58] utilizing the yield surface proposed by Orbison et al. [18] is employed to capture partial plasticization effects.The shear deformation stiffness matrix proposed by Chen et al. [19] is also used to consider the effects of transverse shear deformation.An initial story out-of-plane considering initial imperfection is applied to account for global system imperfections.The generalized displacement control (GDC) method [20] is employed to solve the nonlinear equations.
In this study the performance of ML algorithms is evaluated using two measures of mean square error (MSE) as follows: Where N is the number of samples; y i and y i ' are the original and predicted values of the i th output, respectively; and is the average of all the original output data

Method to create data set for ML
Regarding the regression load-bearing capacity of nonlinear steel frames presented in this study, the input is limited to the properties of the cross-sections of beams and columns.A total of sixteen features of the W-section are considered.These characteristics are input to perform advanced analysis as described in the previous section.the output for the regression problem is the ULF of the structural.The steps to create a dataset are as follows: Step 01: Determine the shape and material of the frame and the applied load.
Step 02: Determine the number of data samples and the number of cross-sections included in a section design group of the structure.
Step 03: Generate M random samples (X 1 , X 2 ...X M ) design group of structural sections (X 1 , X 2 ,..X N ), in which X i selects the section for i th element group.
Step 04: Calculate the final load factor lf i corresponding to sample X i using PAAP.
Step 05: Determine the corresponding input and output of the sample i th based on X i and lf i Step 06: Save the data.

Case study
Figs. 2 and 3 show the layout and geometry of the 20-story space frame.In this frame, 260 beams and 200 columns are categorized into 10 groups and 10 groups respectively.Therefore, there are 320 input variables considered in this example.W14 and W16 are specified to design column groups 1 to 5; W12 and W14 are used for column groups 6 to 10, and W10, W12, W14, and W16 are used for beams.Hence, there are about 1.91E+37 permutations in this case study.An initial geometry imperfection of 1/500 is applied for the column members to both X-and Y-axes.The applied loads include wind loads of 0.29 kN/m 2 acting in the Y-direction and gravity loads of 2.4 kN/m 2 .

Fig. 2. Plan view and model view of 20_story space steel frame
From Fig. 3., it appears that XGBoost has the best results of average MSE among cases, while LR has the worst results.It is noteworthy that XGBoost and RF have very high accuracy in the case of small N t (average MSEs of XGBoost and RF are 0.000792 and 0.00124, respectively at N t equal to 1,000).At the final iteration, the XGBoost, RF, and DL methods provide very high accuracy.Moreover, DL provides the second worse results in a small number of samples but when N t takes values to 20,000 it is much better in prediction.This shows that DL has the best convergence rate and can catch the other ML methods at the final iteration although it starts with the high value of average MSE at early iterations; while SVM has the lowest convergence rate.XGBoost is the most efficient method for regression of the ULF of the twenty-story space frame.The summary of MSE and time effort for each ML algorithm is shown in Table 2.

Conclusion
Based on observations made in the case study, it can be concluded that XGBoost provides the best performance in the regression of ULF regardless of the number of samples.LR shows the least accuracy but cost the least time-consuming because of its simplicity.In addition, XGBoost and RF have high accuracy in the regression of ULF.Besides, DL has the highest convergence rate, while SVM has the lowest convergence rate for the regression of ULF regardless of N t .It also needs to be added that the robustness of ML algorithms in one application does not necessarily guarantee their success and applicability in providing accurate response predictions in others.This is only one case study of the space frame, there needs to be more research and investigation of the performance of differential ML techniques to account for the complexity level of the frame and the number of design variables to fulfill this conclusion.Investigating efficient ML methods that can provide approximations to real structural responses not only expands applications in engineering analysis and design but also enables cost-effective safety assessments and lifecycle management of the structure.

Fig. 3 .
Fig. 3. Performance of ML algorithms in ULF regression for 20-story space frame.

Table 1 .
Properties of W-section (Input for meta-model)

No. Cross-section properties No. Cross-section properties
Due to the large number of samples in the database, the training dataset and the test dataset are to be developed from the database.The number of samples for the test dataset was fixed at 5000, and the number of samples for the selected training dataset was 1000, 2000, 5000, 10000, and 20000.All samples in the training and test datasets were the same.

Table 2 .
Performance of 5 regression techniques in prediction of ULF