Research on application system log anomaly detection based on federated transfer learning

Abstract:

Significant progress has been made in the research of log anomaly detection. However, two challenges still exist in reality. Firstly, log data is often stored on different servers, creating "data islands", the number of abnormal samples in the log data of a single company or organization is insufficient and the abnormal patterns are relatively limited, it is a challenge to train a detection model with high accuracy through these data. Integrating log data from different sources can improve the model''s performance but may result in log data leakage during transmission; Secondly,the log data of different application system types varies in log structure and syntax, and simple integration for training models is ineffective. To address these issues, this paper proposes a log anomaly detection training framework called LogFTL based on federated transfer learning, which uses federated learning algorithm based on matching average. On the premise of ensuring the privacy and security of the client''s data, LogFTL aggregates the model parameters of the client on the server side to form a global model which is then distributed to the client side. Using the client''s local data, the LogFTL framework migrates and learns to optimize the client’s local model and the detection effect of local log data is improved.The experiment resluts show that the LogFTL framework proposed in this paper outperforms traditional log anomaly detection methods in federated learning scenarios, and demonstrate the transfer learning effectiveness of LogFTL.