• Student: OliverPinter (op@)
• Mentor: GavinAtkinson (gavin@)

In first phase, I want to implement the Intel SMAP (Supervisor Mode Access Prevention) technology for x86-64 architecture. In second phase, I plan to implement boot/load time kernel and kernel module patching framework.

Intel SMAP is a hardware extension to support advanced kernel self-protection. The SMAP technology will prevent unintended data access from kernel to userland memory. The technology will appear in Intel Broadwell architecture in 2014Q2/Q3. Currently there is an emulator - namely Qemu with TCG - which supports this technology.

Runtime kernel/kernel module patching is required, otherwise the processor will fail when processing unknown instruction. Newer processors introducing newer instructions which didn't exist on older one. To solve this situation this framework makes the kernel and kernel modules self-modifiable in common way.

phase #1: - Improved security of FreeBSD kernel in future x86-64 processors

phase #2: - generic framework for boot-time/runtime kernel image and kernel modules patching - elliminate hackish “manual” instruction patching: http://svnweb.freebsd.org/base/head/sys/amd64/amd64/cpu_switch.S?r1=238450&r2=238449&pathrev=238450

phase #1:

phase #2:

phase #1 - SMAP:

phase #2 - kernel patching:

https://svnweb.freebsd.org/socsvn/soc2014/op/

https://github.com/opntr

https://github.com/opntr/opBSD/tree/op/gsoc2014/master

https://github.com/opntr/opBSD/tree/op/gsoc2014/smap

https://github.com/opntr/opBSD/tree/op/gsoc2014/kpatch

http://forums.grsecurity.net/viewtopic.php?f=7&t=3046

https://lwn.net/Articles/517475/

http://www.freebsd.org/doc/en_US.ISO8859-1/articles/committers-guide/subversion-primer.html

http://www.intel.com/content/dam/www/public/us/en/documents/manuals/64-ia-32-architectures-software-developer-manual-325462.pdf

https://software.intel.com/sites/default/files/managed/68/8b/319433-019.pdf

http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2012/10/24592_APM_v11.pdf

http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2012/10/24593_APM_v21.pdf

http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2008/10/24594_APM_v3.pdf

http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2012/10/26568_APM_v41.pdf

http://amd-dev.wpengine.netdna-cdn.com/wordpress/media/2012/10/26569_APM_v51.pdf

http://tigcc.ticalc.org/doc/gnuasm.html#SEC109

http://llvm.org/docs/doxygen/html/AsmParser_8cpp_source.html

http://clang.llvm.org/compatibility.html

http://www.ibm.com/developerworks/aix/library/au-gdb.html

https://sourceware.org/binutils/docs/ld/Scripts.html#Scripts

319433-019.pdf : 10.3; table 2-5; page 887; page 894

CPUID.(EAX=07H,ECX=0H):EBX.SMAP[bit 20] = 1

stac:

STAC sets the AC flag bit in EFLAGS/RFLAGS without affecting other bits. Attempt to execute STAC when CPL > 0 will cause #UD. This instruction's operation is the same in non-64-bit modes and 64-bit mode.

clac:

CLAC clears the AC flag bit in EFLAGS/RFLAGS without affecting other bits. Attempt to execute CLAC when CPL > 0 will cause #UD. This instruction's operation is the same in on-64-bit modes and 64-bit mode.

EFLAGS.AC

“System software enables SMAP by setting the SMAP flag in control register CR4 (bit 21).”

“If CPL < 3, SMAP protections are disabled if EFLAGS.AC = 1. If CPL = 3, SMAP applies to all supervisor-mode data accesses (these are implicit supervisor accesses) regardless of the value of EFLAGS.AC.” - Intel Ref. # 319433-014 9.3.2

BOOL
check_smap_access_violation(FRAME, USERMODE)
{
 if SMAP_DISABLED :
  RETURN FALSE;

 if  CPL == 3 OR RFLAGS@FRAME BITAND EFLAGS.AC :
  RETURN FALSE;

 RETURN TRUE;
}

• i386 part originally not part of this project
• i386 part currently not tested
• do NOT compile the kernel with CPUTYPE=core-avx2

• potential useable kernel API's:
  • linker_file_lookup_set(…)
  • pmap_enter(…)
  • pmap_kenter(…)
  • sys/linker_set.h
  • CPU rendezvous

• basic structures

• lf_selfpatch_t

typedef struct lf_selfpatch {
        void    *patchable;           /* address of patchable instractions */
        void    *patch;               /* address of patch */
        short   feature_selector;
        int     feature;
        char    *comment;
} lf_selfpatch_t;

• elf sections

• selfpatch_metainfo

• selfpatch_patch

• assembly instruction size calculation like in https://github.com/freebsd/freebsd/blob/master/sys/amd64/include/asmacros.h#L217

• patching scenarios

• kernel

→ hammer_time

→ lf_selfpatch(linker_kernel_file)

• module

→ modload

→ lf_selfpatch(linker_module_file)

• module → kernel (patching the kernel image, from newly loaded kernel modules, probably one way to apply security patches run-time)

→ onload

→ lf_patch(linker_module_file, linker_destination_file)

→ smp_rendezvous_cpu(NULL, lf_selfpatch(foo), NULL, arg)

• implemented kernel image patching on boot

• implemented module patching at preload and on load

• tested kernel image patching on boot

• updated build tools and maintaining tools

• see commits in https://github.com/opntr/opBSD/tree/op/gsoc2014/kpatch

• fast test:

op@pandora-d tools> ./run_smap_test_vm.csh
Triple fault
CPU Reset (CPU 0)
RAX=00000000f000ff00 RBX=0000000000000001 RCX=0000000000000000 RDX=ffffffff80d0daa0
RSI=ffffffff800c4790 RDI=ffffffff80044100 RBP=ffffffff800440f0 RSP=ffffffff80043ef0
R8 =ffffffff80d0db80 R9 =0000000000000000 R10=0000000000000020 R11=0000000000000000
R12=ffffffff818bc8c8 R13=ffffffff8093beb0 R14=ffffffff80044100 R15=ffffffff818bcd90
RIP=ffffffff80d0d94f RFL=00000297 [--S-APC] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0028 0000000000000000 ffffffff 00af9300 DPL=0 DS   [-WA]
CS =0020 0000000000000000 ffffffff 00af9b00 DPL=0 CS64 [-RA]
SS =0028 0000000000000000 ffffffff 00af9300 DPL=0 DS   [-WA]
DS =0028 0000000000000000 ffffffff 00af9300 DPL=0 DS   [-WA]
FS =0028 0000000000000000 ffffffff 00af9300 DPL=0 DS   [-WA]
GS =0028 ffffffff818d7d80 ffffffff 00af9300 DPL=0 DS   [-WA]
LDT=0000 0000000000000000 0000ffff 00008200 DPL=0 LDT
TR =0038 0000000000005f98 00002067 00008900 DPL=0 TSS64-avl
GDT=     ffffffff818d6350 00000067
IDT=     ffffffff8184b4c0 00000fff
CR0=80000011 CR2=0000000000005fbc CR3=0000000000044000 CR4=00000030
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=0000000000000013 CCD=ffffffffffffffee CCO=SUBL
EFER=0000000000000500
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000
XMM08=00000000000000000000000000000000 XMM09=00000000000000000000000000000000
XMM10=00000000000000000000000000000000 XMM11=00000000000000000000000000000000
XMM12=00000000000000000000000000000000 XMM13=00000000000000000000000000000000
XMM14=00000000000000000000000000000000 XMM15=00000000000000000000000000000000
CPU Reset (CPU 0)
EAX=00000032 EBX=00000000 ECX=00000000 EDX=00000097
ESI=0000000a EDI=00100000 EBP=00000000 ESP=00000fb8
EIP=000f13d3 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0010 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
CS =0008 00000000 ffffffff 00cf9b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
DS =0010 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
FS =0010 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
GS =0010 00000000 ffffffff 00cf9300 DPL=0 DS   [-WA]
LDT=0000 00000000 0000ffff 00008200 DPL=0 LDT
TR =0000 00000000 0000ffff 00008b00 DPL=0 TSS32-busy
GDT=     000f6688 00000037
IDT=     000f66c6 00000000
CR0=60000011 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
CCS=00000000 CCD=00000001 CCO=LOGICL
EFER=0000000000000000
FCW=037f FSW=0000 [ST=0] FTW=00 MXCSR=00001f80
FPR0=0000000000000000 0000 FPR1=0000000000000000 0000
FPR2=0000000000000000 0000 FPR3=0000000000000000 0000
FPR4=0000000000000000 0000 FPR5=0000000000000000 0000
FPR6=0000000000000000 0000 FPR7=0000000000000000 0000
XMM00=00000000000000000000000000000000 XMM01=00000000000000000000000000000000
XMM02=00000000000000000000000000000000 XMM03=00000000000000000000000000000000
XMM04=00000000000000000000000000000000 XMM05=00000000000000000000000000000000
XMM06=00000000000000000000000000000000 XMM07=00000000000000000000000000000000

op@pandora-d opBSD.git> addr2line --exe=/tmp/kernelbuild/boot/kernel/kernel ffffffff80d0d94f
/usr/data/source/git/opBSD/opBSD.git/sys/amd64/amd64/trap.c:219

• http://en.wikibooks.org/wiki/QEMU/Monitor#logfile
• there is the error

Breakpoint 2, getenv_string (name=0xffffffff80f922df "kern.msgbufsize", data=0xffffffff814c8cb0 "", 
    size=258) at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_environment.c:455
455             if (dynamic_kenv) {
(gdb) c
Continuing.

Breakpoint 2, getenv_string (name=0xffffffff80f922ef "kern.maxtsiz", data=0xffffffff814c8cb0 "", 
    size=258) at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_environment.c:455
455             if (dynamic_kenv) {
(gdb) c
Continuing.

Breakpoint 2, getenv_string (name=0xffffffff80f922fc "kern.dfldsiz", data=0xffffffff814c8cb0 "", 
    size=258) at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_environment.c:455
455             if (dynamic_kenv) {
(gdb) c
Continuing.

Breakpoint 2, getenv_string (name=0xffffffff80f92309 "kern.maxdsiz", data=0xffffffff814c8cb0 "", 
    size=258) at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_environment.c:455
455             if (dynamic_kenv) {
(gdb) c
Continuing.

Breakpoint 2, getenv_string (name=0xffffffff80f92316 "kern.dflssiz", data=0xffffffff814c8cb0 "", 
    size=258) at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_environment.c:455
455             if (dynamic_kenv) {
(gdb) c
Continuing.

Breakpoint 2, getenv_string (name=0xffffffff80f92323 "kern.maxssiz", data=0xffffffff814c8cb0 "", 
    size=258) at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_environment.c:455
455             if (dynamic_kenv) {
(gdb) c
Continuing.

Breakpoint 2, getenv_string (name=0xffffffff80f92330 "kern.sgrowsiz", data=0xffffffff814c8cb0 "", 
    size=258) at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_environment.c:455
455             if (dynamic_kenv) {
(gdb) c
Continuing.

Breakpoint 2, getenv_string (name=0xffffffff80f9233e "kern.ngroups", data=0xffffffff814c8cb0 "", 
    size=258) at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_environment.c:455
455             if (dynamic_kenv) {
(gdb) c
Continuing.

Breakpoint 2, getenv_string (name=0xffffffff80f9234b "kern.pid_max", data=0xffffffff814c8cb0 "", 
    size=258) at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_environment.c:455
455             if (dynamic_kenv) {
(gdb) c
Continuing.

Breakpoint 2, getenv_string (name=0xffffffff80f92358 "vfs.unmapped_buf_allowed", 
    data=0xffffffff814c8cd0 "", size=258)
    at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_environment.c:455
455             if (dynamic_kenv) {
(gdb) c
Continuing.

triple fault itt:

hammer_time (modulep=<value optimized out>, physfree=29208576)
    at /usr/data/source/git/opBSD/opBSD.git/sys/amd64/amd64/machdep.c:1898
1898            PCPU_SET(prvspace, pc);
(gdb) 
1899            PCPU_SET(curthread, &thread0);
(gdb) 
1900            PCPU_SET(tssp, &common_tss[0]);
(gdb) 
1901            PCPU_SET(commontssp, &common_tss[0]);
(gdb) 
1902            PCPU_SET(tss, (struct system_segment_descriptor *)&gdt[GPROC0_SEL]);
(gdb) 
1903            PCPU_SET(ldt, (struct system_segment_descriptor *)&gdt[GUSERLDT_SEL]);
(gdb) 
1904            PCPU_SET(fs32p, &gdt[GUFS32_SEL]);
(gdb) 
1905            PCPU_SET(gs32p, &gdt[GUGS32_SEL]);
(gdb) 
1915            mutex_init();
(gdb) 
mutex_init () at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_mutex.c:952
952             init_turnstiles();
(gdb) 
init_turnstiles () at /usr/data/source/git/opBSD/opBSD.git/sys/kern/subr_turnstile.c:364
364                     LIST_INIT(&turnstile_chains[i].tc_turnstiles);
(gdb) 
365                     mtx_init(&turnstile_chains[i].tc_lock, "turnstile chain",
(gdb) 
_mtx_init (c=0xffffffff8155d160, name=0xffffffff80f94b80 "turnstile chain", type=0x0, opts=1)
    at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_mutex.c:871
871             m = mtxlock2mtx(c);
(gdb) 
875             ASSERT_ATOMIC_LOAD_PTR(m->mtx_lock,
(gdb) 
885             if (opts & MTX_SPIN)
(gdb) 
892             if (opts & MTX_RECURSE)
(gdb) 
894             if ((opts & MTX_NOWITNESS) == 0)
(gdb) 
896             if (opts & MTX_DUPOK)
(gdb) 
898             if (opts & MTX_NOPROFILE)
(gdb) 
902             lock_init(&m->lock_object, class, name, type, flags);
(gdb) 
lock_init (lock=0xffffffff8155d148, class=0xffffffff81463730, 
    name=0xffffffff80f94b80 "turnstile chain", type=0x0, flags=131072)
    at /usr/data/source/git/opBSD/opBSD.git/sys/kern/subr_lock.c:78
78              KASSERT(!lock_initialized(lock), ("lock \"%s\" %p already initialized",
(gdb) 
83                      if (lock_classes[i] == class) {
(gdb) 
84                              lock->lo_flags = i << LO_CLASSSHIFT;
(gdb) 
90              lock->lo_name = name;
(gdb) 
91              lock->lo_flags |= flags | LO_INITIALIZED;
(gdb) 
93              WITNESS_INIT(lock, (type != NULL) ? type : name);
(gdb) 
witness_init (lock=0xffffffff8155d148, type=0xffffffff80f94b80 "turnstile chain")
    at /usr/data/source/git/opBSD/opBSD.git/sys/kern/subr_witness.c:809
809             class = LOCK_CLASS(lock);

ITT A HIBA:

0xffffffff809500f0 <witness_init+0>:    push   %rbp
0xffffffff809500f1 <witness_init+1>:    mov    %rsp,%rbp
0xffffffff809500f4 <witness_init+4>:    push   %r15
0xffffffff809500f6 <witness_init+6>:    push   %r14
0xffffffff809500f8 <witness_init+8>:    push   %rbx
0xffffffff809500f9 <witness_init+9>:    push   %rax
0xffffffff809500fa <witness_init+10>:   mov    %rsi,%r14
0xffffffff809500fd <witness_init+13>:   mov    %rdi,%rbx
0xffffffff80950100 <witness_init+16>:   mov    0x8(%rbx),%eax
0xffffffff80950103 <witness_init+19>:   mov    $0x418,%ecx
>>> 0xffffffff80950108 <witness_init+24>:   (bad)  
0xffffffff80950109 <witness_init+25>:   loop   0xffffffff8095017b <witness_init+139>
>>> 0xffffffff8095010b <witness_init+27>:   (bad)  
0xffffffff8095010c <witness_init+28>:   enterq $0x8b4c,$0x3c
0xffffffff80950110 <witness_init+32>:   int    $0xd0
0xffffffff80950112 <witness_init+34>:   stos   %al,%es:(%rdi)
0xffffffff80950113 <witness_init+35>:   rex.RX subl   $0xf6412574,0x80000(%rcx)
0xffffffff8095011e <witness_init+46>:   rex.RXB or     %r9b,(%r8)
0xffffffff80950121 <witness_init+49>:   jne    0xffffffff80950141 <witness_init+81>
0xffffffff80950123 <witness_init+51>:   mov    (%r15),%rdx
0xffffffff80950126 <witness_init+54>:   mov    (%rbx),%rcx
0xffffffff80950129 <witness_init+57>:   mov    $0xffffffff80f95af0,%rdi
0xffffffff80950130 <witness_init+64>:   mov    $0xffffffff80f95b17,%rsi

ffffffff809500f0 <witness_init>:
ffffffff809500f0:       55                      push   %rbp
ffffffff809500f1:       48 89 e5                mov    %rsp,%rbp
ffffffff809500f4:       41 57                   push   %r15
ffffffff809500f6:       41 56                   push   %r14
ffffffff809500f8:       53                      push   %rbx
ffffffff809500f9:       50                      push   %rax
ffffffff809500fa:       49 89 f6                mov    %rsi,%r14
ffffffff809500fd:       48 89 fb                mov    %rdi,%rbx
ffffffff80950100:       8b 43 08                mov    0x8(%rbx),%eax
ffffffff80950103:       b9 18 04 00 00          mov    $0x418,%ecx
ffffffff80950108:       c4                      (bad)  
ffffffff80950109:       e2 70                   loop   ffffffff8095017b <witness_init+0x8b>
ffffffff8095010b:       f7                      (bad)  
ffffffff8095010c:       c8 4c 8b 3c             enterq $0x8b4c,$0x3c
ffffffff80950110:       cd d0                   int    $0xd0

(kgdb) info break
Num Type           Disp Enb Address            What
5   breakpoint     keep y   0xffffffff80e187c6 in atpic_reset at cpufunc.h:260
6   breakpoint     keep y   0xffffffff808b1d00 in cninit_finish
                                               at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_cons.c:162
7   breakpoint     keep y   0xffffffff808b18c0 in cninit
                                               at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_cons.c:109  
        breakpoint already hit 2 times
8   breakpoint     keep y   0xffffffff808b1c74 in cnselect
                                               at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_cons.c:236
(kgdb) b cnremove
Breakpoint 9 at 0xffffffff808b1a40: file /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_cons.c, line 200.
(kgdb) c
Continuing.

Breakpoint 7, cninit () at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_cons.c:109
109     {
(kgdb) c
Continuing.

Breakpoint 9, cnremove (cn=0xffffffff814048d8)
    at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_cons.c:200
200     {
(kgdb) bt
#0  cnremove (cn=0xffffffff814048d8) at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_cons.c:200
#1  0xffffffff808b191b in cninit () at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_cons.c:129
#2  0xffffffff80cf0196 in hammer_time (modulep=<value optimized out>, physfree=<value optimized out>)
    at /usr/data/source/git/opBSD/opBSD.git/sys/amd64/amd64/machdep.c:1961
#3  0xffffffff802da024 in btext () at /usr/data/source/git/opBSD/opBSD.git/sys/amd64/amd64/locore.S:79
#4  0x0000000000000000 in ?? ()
(kgdb) c
Continuing.

Breakpoint 7, cninit () at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_cons.c:109
109     {
(kgdb) c
Continuing.

Breakpoint 9, cnremove (cn=0xffffffff814048d8)
    at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_cons.c:200
200     {
(kgdb) s
204             STAILQ_FOREACH(cnd, &cn_devlist, cnd_next) {
(kgdb) 
229     }
(kgdb) 
cninit () at /usr/data/source/git/opBSD/opBSD.git/sys/kern/kern_cons.c:131
131                     if (cn->cn_ops == NULL)
(kgdb) 
133                     cn->cn_ops->cn_probe(cn);
(kgdb) 
sc_cnprobe (cp=0xffffffff814048d8)
    at /usr/data/source/git/opBSD/opBSD.git/sys/dev/syscons/syscons.c:1592
1592        if (getenv("hw.syscons.disable")) {
(kgdb) ni
0xffffffff80734925      1592        if (getenv("hw.syscons.disable")) {
(kgdb) 
0xffffffff8073492a      1592        if (getenv("hw.syscons.disable")) {
(kgdb) 
0xffffffff8073492d      1592        if (getenv("hw.syscons.disable")) {
(kgdb) 
0xffffffff80734937      1593            cp->cn_pri = CN_DEAD;
(kgdb) 
0xffffffff8073493b      1593            cp->cn_pri = CN_DEAD;
(kgdb) 
1597        cp->cn_pri = sc_get_cons_priority(&unit, &flags);
(kgdb) 

triple fault on 1597 cp→cn_pri = sc_get_cons_priority(&unit, &flags); line this error caused by compiler error, and occurred when in /etc/make.conf set CPUTYPE to core-avx2