Citation :

Khaja Kamaluddin, 2023. "Dynamic Malware Analysis through System Call Tracing and API Monitoring", ESP International Journal of Advancements in Computational Technology (ESP-IJACT)  Volume 1, Issue 3: 167-179.

Abstract :

It is discussed that malware authors are moving towards more advanced techniques to bypass detection. Static methods of analysis, while useful, are limited by obfuscated or polymorphic malware. It has thus become necessary for dynamic malware analysis, or the real-time execution of malicious software and its observation of behaviour. There are two main approaches in this paradigm: system call tracing and API monitoring. These techniques are granular in providing visibility of malware behaviour by logging interactions with the operating system and the core services. This review paper gives a comprehensive analysis of these techniques initially by considering their principles, tools, effectiveness, and limitations. Additionally, it examines recent advancements in the hybrid analysis frameworks that combine both techniques to enhance the detection accuracy. This paper seeks to provide a consolidated reference in terms of real-world case studies, comparative tables, statistical graphs, and insights from more than twenty scholarly sources to those researchers and practitioners who seek to enhance their malware detection capabilities. It also defines future directions of integration to make dynamic malware analysis more robust and scalable, as well as using machine learning, cloud-based analysis, and standardized benchmarks.

References :

[1] K. Rieck, T. Holz, C. Willems, P. Düssel, and P. Laskov, "Learning and classification of malware behaviour," in Detection of Intrusions and Malware, and Vulnerability Assessment, Paris, France: Springer, 2008, pp. 161–182.
[2] S. Kirat, G. Vigna, and C. Kruegel, "BareCloud: Bare-metal analysis-based evasive malware detection," in 23rd USENIX Security Symposium, San Diego, CA, USA, 2014, pp. 287–301.
[3] E. Raff, J. Barker, J. Sylvester, R. Brandon, B. Catanzaro, and C. Nicholas, "Malware detection by eating a whole EXE," in Proceedings of the AAAI Workshop on Artificial Intelligence for Cyber Security, 2018. [Online]. Available: https://arxiv.org/abs/1710.09435
[4] J. Saxe and K. Berlin, "Deep neural network-based malware detection using two-dimensional binary program features," in 10th International Conference on Malicious and Unwanted Software (MALWARE), Fajardo, PR, USA, 2015, pp. 11–20. [Online]. Available: https://arxiv.org/abs/1508.03096
[5] B. Kolosnjaji, A. Zarras, G. Webster, and C. Eckert, "Deep learning for classification of malware system call sequences," in Australasian Joint Conference on Artificial Intelligence, Hobart, Australia: Springer, 2016, pp. 137–149.
[6] S. Mohurle and M. Patil, "A brief study of WannaCry threat: Ransomware attack 2017," International Journal of Advanced Research in Computer Science, vol. 8, no. 5, pp. 1938–1940, 2017.
[7] F. C. Freiling, T. Holz, and G. Wicherski, "Botnet tracking: Exploring a root-cause methodology to prevent distributed denial-of-service attacks," in European Symposium on Research in Computer Security (ESORICS), 2005, pp. 319–335.
[8] U.S. Department of Justice, "Emotet botnet disrupted in international cyber operation," Jan. 2021. [Online]. Available: https://www.justice.gov/archives/opa/pr/emotet-botnet-disrupted-international-cyber-operation
[9] A. Austin and M. Stamp, "Static, dynamic, and hybrid analysis for malware detection," Journal of Computer Virology and Hacking Techniques, vol. 11, no. 2, pp. 95–105, 2015.
[10] Y. Xia, Y. Liu, J. Li, and H. Jin, "Evasion techniques: Future directions for malware analysis," IEEE Access, vol. 7, pp. 63664–63679, 2019.
[11] G. Gu, P. Porras, V. Yegneswaran, M. Fong, and W. Lee, "BotHunter: Detecting malware infection through IDS-driven dialog correlation," in USENIX Security Symposium, 2007.
[12] B. Anderson, D. Quist, J. Neil, C. Storlie, and T. Lane, "Graph-based malware detection using dynamic analysis," Journal in Computer Virology, vol. 7, no. 4, pp. 247–258, 2011.
[13] A. Lanzi, D. Balzarotti, C. Kruegel, M. Christodorescu, and E. Kirda, "AccessMiner: Using system-centric models for malware detection," in ACM Conference on Computer and Communications Security (CCS), 2010, pp. 399–412.
[14] M. Christodorescu and S. Jha, "Testing malware detectors," ACM SIGSOFT Software Engineering Notes, vol. 29, no. 4, pp. 34–44, 2004.
[15] N. Idika and A. P. Mathur, "A survey of malware detection techniques," Purdue University, CERIAS Tech Report, 2007. [Online]. Available: https://www.cerias.purdue.edu/assets/pdf/bibtex_archive/2007-26.pdf
[16] Y. Ye, T. Li, D. Adjeroh, and S. S. Iyengar, "A survey on malware detection using data mining techniques," IEEE Communications Surveys & Tutorials, vol. 19, no. 4, pp. 2175–2216, 2017.
[17] D. Ucci, L. Aniello, and R. Baldoni, "Survey of machine learning techniques for malware analysis," Computers & Security, vol. 81, pp. 123–147, 2019.
[18] A. Mohaisen and O. Alrawi, "Unveiling Zeus: Automated classification of malware samples," in Proceedings of the 22nd International Conference on World Wide Web Companion, 2013, pp. 829–836.
[19] S. E. Schechter and M. D. Smith, "Accessing protected resources through API monitoring," IEEE Security & Privacy, vol. 1, no. 5, pp. 62–65, 2003.
[20] J. Caballero, P. Poosankam, C. Kreibich, and D. Song, "Dispatcher: Enabling active botnet infiltration using dynamic analysis," in RAID, 2009, pp. 41–60.
[21] P. Vinayakumar, K. P. Soman, and P. Poornachandran, "Applying deep learning approaches for network traffic prediction," in International Conference on Advances in Computing, Communications and Informatics (ICACCI), 2017.
[22] S. M. Fadhlullah and H. Hasbullah, "Review on dynamic malware analysis techniques," Australian Journal of Basic and Applied Sciences, vol. 5, no. 10, pp. 500–510, 2011. [Online]. Available: https://ajbasweb.com/old/ajbas/2011/October-2011/500-510.pdf
[23] R. Perdisci, A. Lanzi, and W. Lee, "Behavioral clustering of HTTP-based malware," in USENIX NSDI, 2010.
[24] Dixit, S. (2020). The impact of quantum supremacy on cryptography: Implications for secure financial transactions. International Journal of Scientific Research in Computer Science, Engineering and Information Technology, 6(4), 611–637. https://doi.org/10.32628/CSEIT2064141
[25] Yashu, F., Saqib, M., Malhotra, S., Mehta, D., Jangid, J., & Dixit, S. (2021). Thread mitigation in cloud native application development. Webology, 18(6), 10160–10161. https://www.webology.org/abstract.php?id=5338s

Keywords :

Dynamic Analysis, Malware Detection, System Calls, API Monitoring, Behaviour-based Analysis, Cybersecurity, Sandboxing, Threat Intelligence