Control Flow Enforcement Technology — Intel® SDE documentation

# Control Flow Enforcement Technology

Control Flow Enforcement Technology (Intel® CET), is hardware extension to detect malware that try to gain control over software using return-oriented-programming (ROP), or using indirect jump-oriented-programming (JOP). This feature was introduced in Intel® Tiger Lake CPU.

Intel® SDE provides emulation for Intel® CET technology for checking the readiness of software compiled with Intel® CET. This emulation requires some initialization and emulation work for existing (legacy) instructions and this adds performance overhead even when the emulation is not required. Therefore users need to tell Intel® SDE to activate Intel® CET emulation.

Please note that Intel® Pin, the binary instrumentation system, violates Intel® CET checks in its normal work. Therefore, when running Intel® SDE with Intel® CET compiled application on host that supports Intel® CET, the hardware native checks are disabled by Intel® Pin.

## Running Intel® SDE with Intel® CET application

The basic command line invocation is:

    % <path-to-kit>/sde -tgl -cet -- <app>

This enables only stack-checks and it is checks the correlation between call and return instructions across the entire process. This means that if there are issues in the system libraries, they will be reports. Intel® SDE provides a filtering mechanism to ignore and recover from failures. See below for OS specific considerations.

## Stack Checks

Running Intel® SDE with the -cet knob turns on the stack checks. For each thread SDE allocates a shadow stack at the size of 1 page (4Kb) and sets the top of this page as the SSP (shadow stack pointer). If this size is not enough, then you can use the shadow stack size knob to change it, see the knobs below for the knob name.

Intel® SDE reports the errors it detect to a file. You can instruct Intel® SDE to dump the errors to the standard error or to a file name other than the default.

    % sde -tgl -cet -cet-stderr -- hello

    Control flow error: IP: 0x400764 expected (shadow stack): 0x2aaac06d6c36 got (actual return address): 0x2aaac06ea6d3
    INS: ret

Intel® SDE also provides an option to dump the thread’s call stack at the time of the failure. This makes it easier to debug the problem. The default call stack depth is 10 entries, but you can control it with a knob. When the application is compiled with debug information, the call stack includes source file and source line for each entry in the stack.

    % sde -tgl -cet -cet-stderr -cet-call-stack -- hello

    Control flow error: IP: 0x0000000000400764 expected (shadow stack): 0x00002aaac064dc36 got (actual return address): 0x00002aaac06616d3
    INS: ret

    Call stack:
    # IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
    0# 0x00002aaac064db50 __libc_start_main /lib64/libc.so.6:0x00001eb50
    1# 0x0000000000400520 __libc_start_main@plt<full-path-to-application>/application.exe:0x000000520
    2# 0x0000000000400550 _start <full-path-to-application>/application.exe:0x000000550 at /usr/src/packages/BUILD/glibc-2.11.3/csu/../sysdeps/x86_64/elf/start.S:65

See below in the OS specific information for more details.

## Indirect Branch Checks

Intel® CET indirect branch checks consists of a state machine and a special instruction ‘ENDBRANCH’. This instruction (encoded as a NOP instruction in older CPUs) marks the indirect branch target as safe. It is added by the compiler when compiling for indirect branch tracking (IBT). Since applications can load dynamic libraries which were not compiled with Indirect Branch Tracking (i.e. ‘ENDBRACH’), a special legacy compatibility mode was added.

The ‘ENDBRACH’ legacy compatibility treatment allows a Intel® CET enabled program to check for indirect branches even in this case where not all the dynamic libraries have been compiled accordingly. This legacy compatibility support relies on the system loader and the OS to set the legacy code page bitmap. Intel® SDE provides indirect branch checks even when the system loader and runtime libraries are not enabled with Intel® CET IBT support.

Intel® SDE provides two options to turn on indirect branch checks: include images or exclude images. When you include an image the checks happen only in the address range of the included images, when you exclude images, the checks happen on all the images except for the addresses in the excluded images. You should use the include image option when you compile your executable or dynamic libraries with IBT and you want to check these images. You should use the exclude images option when running with runtime libraries which are already compiled with IBT and you want to exclude only specific runtime libraries. You can use the -cet_endbr_exe knob as a shortcut to the -cet_endbr_include_image and the name of the executable.

The ‘ENDBRANCH’ instruction emulation require setting the Intel® Tiger Lake chip knob (or newer).

    > sde -tgl -cet -cet-stderr -cet-endbr-exe -- hello

    Control flow ENDBRANCH error detected at IP: 0x400550 INS: xor ebp, ebp
    Last branch IP: 0x2aaaaaaabb44 INS: jmp r12

    Control flow ENDBRANCH error detected at IP: 0x400780 INS: mov qword ptr [rsp-0x28], rbp
    Last branch IP: 0x2aabc0a2abc0 INS: call rbx

    Control flow error: IP: 0x400764 expected (shadow stack): 0x2aabc0a2ac36 got (actual return address): 0x2aabc0a3e6d3

    INS: ret

    Control flow ENDBRANCH error detected at IP: 0x400620 INS: cmp byte ptr [rip+0x200659], 0x0
    Last branch IP: 0x2aaaaaab9865 INS: call qword ptr [r12+rax*8]

    Control flow ENDBRANCH error detected at IP: 0x40080c INS: push rbp
    Last branch IP: 0x2aaaaaab9882 INS: call rax

As you can see in the output in this case, Intel® SDE checks for both stack and indirect jumps. The indirect branch checks happen only to the executable, and it found a few places where the indirect jumps where from an external library to the executable. You can see in the report the indirect branch target address and the branch that cause this report.

## Debugging and Raising exceptions

Intel® SDE emulation is focused on developing applications enabled with Intel® CET technology. Therefore, the default action when an issue is detected is to write it to the output file (or standard error), and continue or abort. Intel® SDE can provide a call-stack which can be handy to find the issue, but this is not the same as debugging the issues with a debugger.

Intel® SDE provides transparent application debugging that allow the developer to debug the emulated application as if it is running in the native environment. In this case, when issue is detected, a debugger breakpoint can be triggered. This provides the developer full visibility to the issue.

    > <path-to-kit>/sde -debug -tgl -cet -cet-breakpoint -- <app>

Some applications would like to handle the Intel® CET issues internally by interception the exception. For such applications Intel® SDE can raise exception or signal when detecting issues.

    > <path-to-kit>/sde -tgl -cet -cet-raise -- <app>

## Intel® CET on Linux

There are two option of running Intel® CET application under Intel® SDE on Linux. Running on legacy host with legacy runtime libraries, or running on any host with runtime libraries compiled for CET.

## Running with Intel® CET Enabled Runtime Libraries

GCC and binutils with support for Intel® CET have been upstream to the open source and now available on new Linux distributions like Fedora 34. You can use toolchain that support Intel® CET and build your application with CET support. The compiler command line switch is:

    -fcf-protection=[full|branch|return|none]

              branch - do control-flow instrumentation for indirect branches

              return - do control-flow instrumentation for function returns

              full - alias to specify both branch + return

              none - turn off instrumentation

    -mcet/-mshstk/-mibt are Intel specific option to support CET generation

To run Intel® SDE with the application and the native compiled runtime libraries use:

    > <path-to-kit>/sde -tgl -cet -- <app> ...

This means that the loader is parsing the executable and the shared objects and see if they have been compiled with Intel® CET checks. It is using the data inside the ELF image (in the note section) to instruct Intel® SDE to do stack checks, indirect branch checks and set the legacy image information. The loader is calling the SYS_arch_prctl system call to check if the kernel is enabled with Intel® CET. Intel® SDE intercepts the system call and emulate it as if the kernel and the host support Intel® CET checks. Now the loader knows that it can run in Intel® CET enabled mode. In this mode Intel® CET compatible functionality in GLIBC is used.

## Running with Legacy Runtime

Issues and considerations when running with legacy runtime on Linux:

- Running 64 bits applications with only stack checks has problems only with C++ exceptions and setjmp/longjmp use.

- Running 32 bits applications with only stack checks has problems with the way the dynamic loader resolves external symbols (i.e. calls between images). To bypass these problems you need to set the environment variable LD_BIND_NOW. As in (this does not resolve issues with C++ exceptions and setjmp/longjmp):

      > <path-to-kit>/sde -env LD_BIND_NOW 1 -cet -- <app> ...

- Running applications with indirect branch checks (both 32 and 64 bits) suffers from a few issues that we detected, even when checking only the executable and it was compiled with Intel® CET checks. The main issues are calls from the CRT files (init and fini calls). Here is an example of running “hello world” applica

      > sde -cet -cet-stderr -cet-endbr-exe -cet-call-stack -- hw64
      Control flow ENDBRANCH error detected at IP: 0x0000000000400410 INS: xor ebp, ebp
      Last branch IP: 0x00002b27ba199c74 INS: jmp r12
      Call stack:
      # IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
      0# 0x0000000000400410 _start <path-to-exe>/hw64:0x000000410

      Control flow ENDBRANCH error detected at IP: 0x0000000000400500 INS: push r15
      Last branch IP: 0x00002b28cf74f7bd INS: call rbp
      Call stack:
      # IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
      0# 0x0000000000400500 __libc_csu_init <path-to-exe>/hw64:0x000000500
      1# 0x00002b28cf74f7bf __libc_start_main /lib/x86_64-linux-gnu/libc.so.6:0x0000207bf
      2# 0x00002b28cf74f740 __libc_start_main /lib/x86_64-linux-gnu/libc.so.6:0x000020740
      3# 0x00000000004003f0 __libc_start_main@plt<path-to-exe>/hw64:0x0000003f0
      4# 0x0000000000400410 _start <path-to-exe>/hw64:0x000000410

      Hello
      Control flow ENDBRANCH error detected at IP: 0x0000000000400574 INS: sub rsp, 0x8
      Last branch IP: 0x00002b27ba1a9e03 INS: call rax
      Call stack:
      # IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
      0# 0x0000000000400574 _fini <path-to-exe>/hw64:0x000000574
      1# 0x00002b27ba1a9e05 _dl_rtld_di_serinfo /lib64/ld-linux-x86-64.so.2:0x000010e05
      2# 0x00002b27ba1a9b00 _dl_rtld_di_serinfo /lib64/ld-linux-x86-64.so.2:0x000010b00
      3# 0x00002b27ba1a9ab0 _dl_rtld_di_serinfo /lib64/ld-linux-x86-64.so.2:0x000010ab0
      4# 0x00002b28cf768fd8 __libc_secure_getenv /lib/x86_64-linux-gnu/libc.so.6:0x000039fd8
      5# 0x00002b28cf768f10 __libc_secure_getenv /lib/x86_64-linux-gnu/libc.so.6:0x000039f10
      6# 0x00002b28cf769030 exit /lib/x86_64-linux-gnu/libc.so.6:0x00003a030
      7# 0x00002b28cf74f740 __libc_start_main /lib/x86_64-linux-gnu/libc.so.6:0x000020740
      8# 0x00000000004003f0 __libc_start_main@plt<path-to-exe>/hw64:0x0000003f0
      9# 0x0000000000400410 _start <path-to-exe>/hw64:0x000000410

See below for the Intel® CET special knobs.

## Intel® CET on Windows

New version of Windows OS come with Intel® CET enabled runtime libraries. Intel® SDE was not enabled (yet) to run on Windows with enabled runtime libraries. Therefore, Intel® SDE only supports running on legacy mode with limiting the checks. Running small console application (on Win10) might result in multiple error messages even when only doing stack checks. For example:

    > sde -tgl -cet -cet-stderr -cet-call-stack -- hello.exe

    Control flow error: IP: 0x7ff92fff0523 expected (shadow stack): 0x7ff959039600 got (actual return address): 0x7ff95900a052
    INS: ret
    Call stack:
    # IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
    0# 0x7ff959039600 NtTestAlert C:\WINDOWS\SYSTEM32\ntdll.dll:0x0000a9600

    Could not unwind to previous frame: IP: 0x00007ff92fff0523 INS: ret
    Call stack:
    # IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
    0# 0x7ff959039600 NtTestAlert C:\WINDOWS\SYSTEM32\ntdll.dll:0x0000a9600

    hello

Intel® SDE can exclude images also from stack checks, this is useful when there are stack check violations in system libraries.

    > <path-to-kit>/sde -tgl -cet -cet-stderr -cet-exclude-image ntdll.dll -- <app>

When building applications with Intel compiler and with full control flow protection (shadow stack and indirect branch tracking), the compiler adds the ‘ENDBRANCH’ instructions only to the code compiled from the sources. But when linking the object files into an executable or DLL, the linker brings code from CRT files (init/fini code). Since this code has unsafe indirect jumps, you will get errors when running your application in this mode. Please note that you also need to disable incremental linking which also use unsafe indirect jumps.

When running Intel® SDE with Intel® CET mode with checking the executable, you might get errors like the following:

    > sde -tgl -cet -cet-call-stack -cet-stderr -cet-endbr-exe -cet-exclude-image ntdll.dll -- app.exe

    Control flow ENDBRANCH error detected at IP: 0x00007ff76e5014ac INS: sub rsp, 0x28
    Last branch IP: 0x00007ff956935290 INS: jmp rax
    Call stack:
    # IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
    0# 0x00007ff76e5014ac unnamedImageEntryPoint <path>\app.exe:0x0000014ac
    1# 0x00007ff95691836d BaseThreadInitThunk C:\WINDOWS\System32\KERNEL32.DLL:0x00000836d
    2# 0x00007ff956918350 BaseThreadInitThunk C:\WINDOWS\System32\KERNEL32.DLL:0x000008350
    3# 0x00007ff958ff7093 RtlUserThreadStart C:\WINDOWS\SYSTEM32\ntdll.dll:0x000067093

    Control flow ENDBRANCH error detected at IP: 0x00007ff76e501ff0 INS: ret 0x0
    Last branch IP: 0x00007ff76e501e20 INS: jmp qword ptr [rip+0x44479]
    Call stack:
    # IP FUNCTION IMAGE NAME FILE NAME:LINE:COLUMN
    0# 0x00007ff76e503480 unnamedImageEntryPoint <path>\app.exe:0x000003480
    1# 0x00007ff76e502ba4 unnamedImageEntryPoint <path>\app.exe:0x000002ba4
    2# 0x00007ff76e502290 unnamedImageEntryPoint <path>\app.exe:0x000002290
    3# 0x00007ff76e5016fc unnamedImageEntryPoint <path>\app.exe:0x0000016fc
    4# 0x00007ff76e50129c .text <path>\app.exe:0x00000129c
    5# 0x00007ff95691836d BaseThreadInitThunk C:\WINDOWS\System32\KERNEL32.DLL:0x00000836d
    6# 0x00007ff956918350 BaseThreadInitThunk C:\WINDOWS\System32\KERNEL32.DLL:0x000008350
    7# 0x00007ff958ff7093 RtlUserThreadStart C:\WINDOWS\SYSTEM32\ntdll.dll:0x000067093
    ...

Intel® CET control knobs:

`-cet`

  Enable=1 or Disable=0 Intel(R) Control-flow enforcement emulation [default 0]

`-cet_abort`

  Abort the run on the first control-flow enforcements error [default 0]

`-cet_breakpoint`

  Raise a break point when running under the debugger [default 0]

`-cet_call_next`

  Ignore call to next instruction followed with pop instruction [default 1]

`-cet_call_stack`

  Present call stack with control flow enforcement errors [default 0]

`-cet_call_stack_depth`

  Specify control flow enforcement errors call-stack max depth [default 10]

`-cet_endbr_exclude_image`

  Repeatable knob to specify images to exclude from ENDBRANCH checking

`-cet_endbr_exe`

  Add the main executable to the ENDBRANCH checking [default 0]

`-cet_endbr_include_image`

  Repeatable knob to specify images to enable ENDBRANCH checking

`-cet_exclude_image`

  Repeatable knob to specify images to exclude shadow stack checks

`-cet_linux_mode`

  Set the Linux CET kernel interface starting mode (new/old) [default unknown]

`-cet_output_file`

  File name for control flow enforcement errors [default sde-cet-checker-out.txt]

`-cet_raise`

  Raise an exception when CET error detected [default 0]

`-cet_shadow_stack_size`

  Specify the size of the initial shadow stack [default 4096]

`-cet_stderr`

  Report control flow exception to standard error [default 0]