A COMPARATIVE ANALYSIS OF CONTROL FLOW OBFUSCATION TECHNIQUES TO PROTECT SOFTWARES FROM REVERSE ENGINEERING

Abstract

Code obfuscation software protection is an area of paramount importance, which dynamically changes its significance with regard to modern endeavors to ensure the protection of intellectual property against unauthorized access and reverse engineering. The main benefit of obfuscation, overall, is the massive increase in the effort needed to analyze proprietary binaries, which will aid in deterring bad actors and maintain the integrity of commercial software. However, it has been observed that clear and measurable metrics remain absent which can help developers settle on an optimal combination of obfuscation techniques which provides a reasonable level of protection at moderate performance costs. In particular, the research problem covered by the study is the systematic assessment of the effectiveness of the most prevalent methods of controlflow obfuscation, i.e., flattening, splitting, and virtualization, in terms of their correlation with the complexity of binary analysis. Our methodology is to apply the before mentioned techniques on a simple C program by using the Tigress obfuscation toolkit and then examine the resulting binary file using Cutter GUI. The parameters measured included McCabe Cyclomatic Complexity, Control Flow Graph Size and Instruction Count. The effectiveness of the analysis method is evaluated by looking at the change in complexity of the measurements on the obfuscated variants in comparison with the original and with each other, to achieving the best protection profiles. The findings give the developers a practical insight into the best control-flow obfuscation method that would be most effective in hindering reverse engineering with minimal performance costs.