full text Project_Viandr Geovan_21.K1.0047

i PROJECT REPORT COMPARING MACHINE LEARNING ALGORITHM TO BE USED IN SIGNATURE-BASED MALWARE DETECTION VIANDRA GEOVAN 21.K1.0047 Faculty of Computer Science Soegijapranata Catholic University 2025ii HALAMAN PENGESAHAN Judul Tugas Akhir: : COMPARING MACHINE LEARNING ALGORITHM TO BE USED IN SIGNATURE-BASED MALWARE DETECTION Diajukan oleh : VIANDRA GEOVAN NIM : 21.K1.0047 Tanggal disetujui : 16 Juli 2025 Telah disetujui oleh Pembimbing : HIRONIMUS LEONG S.Kom., M.Kom. Penguji 1 : YONATHAN PURBO SANTOSA S.Kom., M.Sc. Penguji 2 : Y.B. DWI SETIANTO S.T., M.Cs. Penguji 3 : Dr. YULIANTO TEJO PUTRANTO S.T., M.T. Penguji 4 : HIRONIMUS LEONG S.Kom., M.Kom. Ketua program studi : ROSITA HERAWATI S.T., M.I.T. Dekan : Prof. Dr. F. RIDWAN SANJAYA S.E., S.Kom., MS.IEC. Halaman ini merupakan halaman yang sah dan dapat diverifikasi melalui alamat di bawah ini. sintak.unika.ac.id/skripsi/verifikasi/?id=21.K1.0047iii DECLARATION OF AUTHORSHIP (Heading Plain) I, the undersigned: Name : Viandra Geovan ID : 21.K1.0047 declare that this work, titled “COMPARING MACHINE LEARNING ALGORITHM TO BE USED IN SIGNATURE-BASED MALWARE DETECTION”, and the work presented in it is my own. I confirm that: 1. This work was done wholly or mainly while in candidature for a research degree at Soegijapranata Catholic University 2. Where any part of this thesis has previously been submitted for a degree or any other qualification at this University or any other institution, this has been clearly stated. 3. Where I have consulted the published work of others, this is always clearly attributed. 4. Where I have quoted from the work of others, the source is always given. 5. Except for such quotations, this work is entirely my own work. 6. I have acknowledged all main sources of help. 7. Where the work is based on work done by myself jointly with others, I have made clear exactly what was done by others and what I have contributed myself. Semarang, 07, 24, 2025 Viandra Geovan 21.K1.0047iv HALAMAN PERNYATAAN PUBLIKASI KARYA ILMIAH UNTUK KEPENTINGAN AKADEMIS Yang bertanda tangan dibawah ini: Nama : Viandra Geovan Program Studi : Teknik Informatika Fakultas : Ilmu Komputer Jenis Karya : Skripsi Menyetujui untuk memberikan kepada Universitas Katolik Soegijapranata Semarang Hak Bebas Royalti Nonekslusif atas karya ilmiah yang berjudul “COMPARING MACHINE LEARNING ALGORITHM TO BE USED IN SIGNATURE-BASED MALWARE DETECTION”. Dengan Hak Bebas Royalti Nonekslusif ini Universitas Katolik Soegijapranata berhak menyimpan, mengalihkan media/formatkan, mengelola dalam bentuk pangkalan data (database), merawat, dan mempublikasikan tugas akhir ini selama tetap mencantumkan nama saya sebagai penulis / pencipta dan sebagai pemilik Hak Cipta. Demikian pernyataan ini saya buat dengan sebenarnya. Semarang, 07, 24, 2025 Yang menyatakan Viandra Geovan 21.K1.0047v ACKNOWLEDGMENT First of all I would like to thank my parents and my family for allowing me to study at this university and for their continuous support throughout my journey as a student here. I would like to also offer my thanks to all the friend that I had made along my journey as a student and thank them also for their companionship. I also thank god for the life that he has given me from birth to now. For the making of this project I would like to give thanks to Mr. Hironemus Leong for his guidance on the making of this project and making it possible for me to finish it. I would also like to thank my closest friend Ardhito, Devano, Stephen, Ivan, Bimo, Niko and Nathan for advising me on how to continue this project. And special thanks to Ardhito for giving me the most advice as well. This project wouldn't happen without them.vi ABSTRACT In today’s digital era, computer security is one of the most significant worries due to technology and the continual threat from malware. However, traditional signature-based malware detection tends to be good against known threats but it fails in cases related to newly created or unknown malwares. The main objective of this study involves comparing machine learning algorithms to see if it can be used to enhance signature-based malware detection to make it more efficient and robust. This research will therefore aim at providing answers to questions on traditional detection methods and the potential for enhancement by ML as well as determining which ML models are best used for malware detection. The study is focused on developing a method that can combine both methods, by considering common malwares and determining the efficiency of various machine learning techniques such as support vector machines, decision trees and convolutional neural networks. This research methodology involves gathering data from credible sources from the internet followed by data preprocessing steps such as cleaning, normalization, and data splitting. Models are trained and evaluated using metrics like accuracy, precision, recall and F1-score. Legal ethical economic aspects were not included in this study thus only technological advancements were focused. The study concludes that using ML with signaturebased detection significantly improves the system's robustness against evolving malware threats. The findings suggest that ML can predict unknown malware patterns, enhancing traditional detection capabilities and providing a more adaptive solution to cybersecurity challenges. This research offers a detailed methodology for replication and aims to contribute to the development of more effective malware detection systems. Keyword: Malware, Signature-based malware detection, cybersecurity, machine learningvii TABLE OF CONTENTS COVER……………………………………………………………………………………………………. i HALAMAN PENGESAHAN ........................................................................................................ ii DECLARATION OF AUTHORSHIP (Heading Plain) ................................................................iii HALAMAN PERNYATAAN PUBLIKASI KARYA ILMIAH UNTUK KEPENTINGAN AKADEMIS .................................................................................................................................. iv ACKNOWLEDGMENT................................................................................................................. v ABSTRACT................................................................................................................................... vi TABLE OF CONTENTS.............................................................................................................. vii LIST OF FIGURE.......................................................................................................................... ix LIST OF TABLE............................................................................................................................ x CHAPTER 1 INTRODUCTION .............................................................................................. 1 1.1 Background........................................................................................................................... 1 1.2 Problem Formulation ........................................................................................................... 2 1.3 Scope..................................................................................................................................... 3 1.4 Objective ............................................................................................................................... 4 CHAPTER 2 LITERATURE STUDY...................................................................................... 5 CHAPTER 3 RESEARCH METHODOLOGY........................................................................ 9 3.1 Research Methodology ....................................................................................................... 9 3.2 Data Collection ................................................................................................................. 10 3.3 Data Preprocessing............................................................................................................ 12 3.3.1 Data Cleaning....................................................................................................... 14 3.3.2 Normalization ...................................................................................................... 14 3.3.3 Data Splitting ....................................................................................................... 15 3.4 Model Selection ................................................................................................................ 16 3.4.1 Decision Tree ..................................................................................................... 16 3.4.2 Support Vector Machines .................................................................................. 17 3.4.3 Random Forest................................................................................................... 19 3.4.4 Convolutional Neural Networks.................................................................................. 20 3.4.5 Criteria For Selection......................................................................................... 21viii 3.5 Model Training ................................................................................................................. 22 3.5.1 Hyperparameter Tuning .................................................................................. 23 3.6 Evaluation ......................................................................................................................... 24 3.6.1 Metrics ......................................................................................................................... 25 CHAPTER 4 IMPLEMENTATION AND RESULTS........................................................... 27 4.1 Experiment Setup................................................................................................................ 27 4.2 Implementation ................................................................................................................... 27 4.2.1 Data Collection ............................................................................................................ 28 4.2.2 Data Cleaning............................................................................................................... 29 4.2.3 Normalizing ................................................................................................................. 29 4.2.4 Data Splitting ............................................................................................................... 30 4.2.5 Decision Tree ............................................................................................................... 30 4.2.6 Support Vector Machines ............................................................................................ 32 4.2.7 Random Forest............................................................................................................. 33 4.2.8 Convolutional Neural Network.................................................................................... 34 4.3 Result .................................................................................................................................. 37 4.3.1 Duplicates .................................................................................................................... 40 4.4 Discussion........................................................................................................................... 43 CHAPTER 5 CONCLUSION................................................................................................. 45 REFERENCES ............................................................................................................................. 47 APPENDIX......................................................................................................................................aix LIST OF FIGURE Figure 3.1 Research Methodology.................................................................................................. 9 Figure 3.2. Preprocessing Diagram............................................................................................... 12 Figure 3.3. Decision Tree diagram ............................................................................................... 17 Figure 3.4. Support vector machines diagram .............................................................................. 18 Figure 3.5. Random Forest visualization ...................................................................................... 20 Figure 4.1. Average Models Result .............................................................................................. 39 Figure 4.2. Average Models Result With Duplicates................................................................... 42x LIST OF TABLE Table 3.1. Data Sample................................................................................................................. 11 Table 3.2. Example of data after data cleaning............................................................................. 13 Table 3.3. Example of data after Normalization........................................................................... 14 Table 3.4. Sample data after normalization .................................................................................. 15 Table 3.5. Data Splitting ............................................................................................................... 15 Table 3.6. Machine Learning Algorithms..................................................................................... 22 Table 3.7. Hyperparameter tuning ................................................................................................ 23 Table 4.1. Decision Tree Result.................................................................................................... 37 Table 4.2. Support Vector Machines Result ................................................................................. 38 Table 4.3. Random Forest Result.................................................................................................. 38 Table 4.4. Convolutional Neural Network Result ........................................................................ 39 Table 4.5. Decision Tree Result Without Data Cleaning.............................................................. 40 Table 4.6. SVM Result Without Data Cleaning ........................................................................... 41 Table 4.7. Random Forest Result Without Data Cleaning............................................................ 41 Table 4.8. CNN Result Without Data Cleaning............................................................................ 42