问题现象
- 复现步骤
- 手机(Android 8.0) + kernel版本4.4
- 操作过程
- 刷入userdebug版本
- 首次启动，低概率出现dex2oat crash
定位分析
- 相关log
- tombstone
Revision: '0' ABI: 'arm' pid: 1122, tid: 1122, name: dex2oat >>> /system/bin/dex2oat <<< signal 6 (SIGABRT), code -6 (SI_TKILL), fault addr -------- Abort message: 'heap.cc:386] Check failed: non_moving_space_mem_map != nullptr Failed to mmap at expected address, mapped at 0xac200000 instead of 0x0ec00000 : Requested region 0x0ec00000-0x12c00000 overlaps with existing map 0x0fa73000-0x0fa92000 (/system/bin/dex2oat)' r0 00000000 r1 00000462 r2 00000006 r3 00000008 r4 00000462 r5 00000462 r6 be8250f0 r7 0000010c r8 0000000d r9 be825151 sl 0000000a fp be825150 ip 00000000 sp be8250e0 lr b0860417 pc b089097c cpsr 200f0010 d0 2e656d69746e7572 d1 2d72203030303166 d2 65747379732f2065 d3 696c2f6e69622f72 d4 3a752f5f5f736569 d5 725f7463656a626f d6 746c75616665643a d7 30733a706f72705f d8 0000000000000000 d9 0000000000000000 d10 0000000000000000 d11 0000000000000000 d12 0000000000000000 d13 0000000000000000 d14 0000000000000000 d15 0000000000000000 d16 2e74736973726570 d17 2e6761742e676f6c d18 000c54b4b00ba000 d19 0000000000000000 d20 0000000800000008 d21 0000000800000008 d22 0000000400000004 d23 0000000400000004 d24 0000238100000381 d25 0000638100004381 d26 0000238000000380 d27 0000638000004380 d28 0000000000000000 d29 0000000000000000 d30 0000000000000000 d31 0000000000000000 scr 30000011 backtrace: #00 pc 0004a97c /system/lib/libc.so (tgkill+12) #01 pc 0001a413 /system/lib/libc.so (abort+54) #02 pc 003385ff /system/lib/libart.so (_ZN3art7Runtime5AbortEPKc+230) #03 pc 00338c8f /system/lib/libart.so (_ZN3art7Runtime7AborterEPKc+10) #04 pc 003efacd /system/lib/libart.so (_ZN7android4base10LogMessageD1Ev+456) #05 pc 0018f14b /system/lib/libart.so (_ZN3art2gc4HeapC1EjjjjddjjRKNSt3__112basic_stringIcNS2_11char_traitsIcEENS2_9allocatorIcEEEENS_14InstructionSetENS0_13CollectorTypeESC_NS0_5space20LargeObjectSpaceTypeEjjjbjjbbbbbbbbbbby+7258) #06 pc 0033a655 /system/lib/libart.so (_ZN3art7Runtime4InitEONS_18RuntimeArgumentMapE+6460) #07 pc 00338cc7 /system/lib/libart.so (_ZN3art7Runtime6CreateEONS_18RuntimeArgumentMapE+50) #08 pc 00014027 /system/bin/dex2oat (_ZN3art7Dex2Oat13CreateRuntimeEONS_18RuntimeArgumentMapE+34) #09 pc 0000b29f /system/bin/dex2oat (_ZN3art7Dex2Oat5SetupEv+606) #10 pc 000080b3 /system/bin/dex2oat (main+514) #11 pc 00017d35 /system/lib/libc.so (__libc_init+48) #12 pc 00007dd0 /system/bin/dex2oat (_start+96)
- tombstone
初步分析
-
问题的直接原因是dex2oat直接abort, 如下:
Abort message: 'heap.cc:386] Check failed: non_moving_space_mem_map != nullptr Failed to mmap at expected address, mapped at 0xac200000 instead of 0x0ec00000 : Requested region 0x0ec00000-0x12c00000 overlaps with existing map 0x0fa73000-0x0fa92000 (/system/bin/dex2oat)' -
相关code
Heap::Heap() { ... ... 374 if (separate_non_moving_space) { 375 ScopedTrace trace2("Create separate non moving space"); 376 // If we are the zygote, the non moving space becomes the zygote space when we run 377 // PreZygoteFork the first time. In this case, call the map "zygote space" since we can't 378 // rename the mem map later. 379 const char* space_name = is_zygote ? kZygoteSpaceName : kNonMovingSpaceName; 380 // Reserve the non moving mem map before the other two since it needs to be at a specific 381 // address. 382 non_moving_space_mem_map.reset( 383 MemMap::MapAnonymous(space_name, requested_alloc_space_begin, 384 non_moving_space_capacity, PROT_READ | PROT_WRITE, true, false, 385 &error_str)); 386 CHECK(non_moving_space_mem_map != nullptr) << error_str; ... ... }这里check fail的原因是MemMap::MapAnonymous()返回了nullptr．
-
MemMap::MapAnonymous()相关code
MemMap* MemMap::MapAnonymous(const char* name, uint8_t* expected_ptr, size_t byte_count, int prot, bool low_4gb, bool reuse, std::string* error_msg, bool use_ashmem) { ... ... void* actual = MapInternal(expected_ptr, page_aligned_byte_count, prot, flags, fd.get(), 0, low_4gb); saved_errno = errno; if (actual == MAP_FAILED) { if (error_msg != nullptr) { if (kIsDebugBuild || VLOG_IS_ON(oat)) { PrintFileToLog("/proc/self/maps", LogSeverity::WARNING); } *error_msg = StringPrintf("Failed anonymous mmap(%p, %zd, 0x%x, 0x%x, %d, 0): %s. " "See process maps in the log.", expected_ptr, page_aligned_byte_count, prot, flags, fd.get(), strerror(saved_errno)); } return nullptr; } if (!CheckMapRequest(expected_ptr, actual, page_aligned_byte_count, error_msg)) { return nullptr; } ... ... }根据代码逻辑，在调用CheckMapRequest()后返回false.
-
CheckMapRequest()相关code
static bool CheckMapRequest(uint8_t* expected_ptr, void* actual_ptr, size_t byte_count, std::string* error_msg) { // Handled first by caller for more specific error messages. CHECK(actual_ptr != MAP_FAILED); if (expected_ptr == nullptr) { return true; } uintptr_t actual = reinterpret_cast<uintptr_t>(actual_ptr); uintptr_t expected = reinterpret_cast<uintptr_t>(expected_ptr); uintptr_t limit = expected + byte_count; if (expected_ptr == actual_ptr) { return true; } // We asked for an address but didn't get what we wanted, all paths below here should fail. int result = munmap(actual_ptr, byte_count); if (result == -1) { PLOG(WARNING) << StringPrintf("munmap(%p, %zd) failed", actual_ptr, byte_count); } if (error_msg != nullptr) { // We call this here so that we can try and generate a full error // message with the overlapping mapping. There's no guarantee that // that there will be an overlap though, since // - The kernel is not *required* to honor expected_ptr unless MAP_FIXED is // true, even if there is no overlap // - There might have been an overlap at the point of mmap, but the // overlapping region has since been unmapped. std::string error_detail; CheckNonOverlapping(expected, limit, &error_detail); std::ostringstream os; os << StringPrintf("Failed to mmap at expected address, mapped at " "0x%08" PRIxPTR " instead of 0x%08" PRIxPTR, actual, expected); if (!error_detail.empty()) { os << " : " << error_detail; } *error_msg = os.str(); } return false; }这里代码很简单，判断expected_ptr和actual_ptr
- 相等: 返回true
- 不相等: 返回false, 并将error message保存
- 根据代码逻辑，初步总结如下:
- 进程要将dalvik-non moving space映射到指定的虚拟地址0ec00000-12c00000(起始地址为300 * MB - 64 * MB, 大小为64MB)
- 实际mmap到0xac200000, 原因是dex2oat占用了0x0fa73000-0x0fa92000, 产生了冲突．
0fa73000-0fa91fff r-x 0 1f000 /system/bin/dex2oat (BuildId: 98d4fb6e838289e7a2bc42b4caf1d455) 0fa93000-0fa93fff r-- 1f000 1000 /system/bin/dex2oat 0fa94000-0fa94fff rw- 0 1000 - 最后抛出错误信息
Abort message: 'heap.cc:386] Check failed: non_moving_space_mem_map != nullptr Failed to mmap at expected address, mapped at 0xac200000 instead of 0x0ec00000 : Requested region 0x0ec00000-0x12c00000 overlaps with existing map 0x0fa73000-0x0fa92000 (/system/bin/dex2oat)'
对比实验
-
使用同一机器旧版本测试，发现dex2oat是映射到高地址
0ec00000-0ec01000 rw-p 00000000 00:05 12697 /dev/ashmem/dalvik-non moving space (deleted) 0ec01000-12c00000 ---p 00001000 00:05 12697 /dev/ashmem/dalvik-non moving space (deleted) 12c00000-1ac00000 rw-p 00000000 00:05 12700 /dev/ashmem/dalvik-main space (region space) (deleted) 8cd23000-8cd42000 r-xp 00000000 103:0b 287 /system/bin/dex2oat 8cd43000-8cd44000 r--p 0001f000 103:0b 287 /system/bin/dex2oat ... ... -
Nexus 6p(OPR6.170623.013)也是将可执行程序映射到高地址(kernel 3.10版本)
... ... 727d3000-727da000 r--p 00000000 103:0b 1108 /system/framework/arm/boot-android.hidl.manager-V1.0-java.oat 727da000-727db000 rw-p 00000000 00:04 450565 /dev/ashmem/dalvik-Zero-initialized segment 3 of ELF file /data/dalvik-cache/arm/system@framework@boot-android.hidl.manager-V1.0-java.oat (deleted) 727db000-727dc000 r--p 00007000 103:0b 1108 /system/framework/arm/boot-android.hidl.manager-V1.0-java.oat 727dc000-727dd000 rw-p 00008000 103:0b 1108 /system/framework/arm/boot-android.hidl.manager-V1.0-java.oat 727dd000-729a2000 rw-p 00000000 00:04 450566 /dev/ashmem/dalvik-non moving space (deleted) 729a2000-767dd000 ---p 001c5000 00:04 450566 /dev/ashmem/dalvik-non moving space (deleted) ab1d5000-ab1f4000 r-xp 00000000 103:0b 342 /system/bin/dex2oat ab1f5000-ab1f6000 r--p 0001f000 103:0b 342 /system/bin/dex2oat ... ... -
现在可以得出初步结论
可执行程序在最近版本的映射，由高地址变为低地址，直接导致了问题的发生．
深入分析
- 经过版本排查，发现问题跟最近kernel小版本升级有关．
- 相关patch如下
commit d2471b5e84f32de4e09b58f5436a4ce3ee935e32 Author: Kees Cook <keescook@chromium.org> Date: Mon Jul 10 15:52:40 2017 -0700 arm: move ELF_ET_DYN_BASE to 4MB commit 6a9af90a3bcde217a1c053e135f5f43e5d5fafbd upstream. Now that explicitly executed loaders are loaded in the mmap region, we have more freedom to decide where we position PIE binaries in the address space to avoid possible collisions with mmap or stack regions. 4MB is chosen here mainly to have parity with x86, where this is the traditional minimum load location, likely to avoid historically requiring a 4MB page table entry when only a portion of the first 4MB would be used (since the NULL address is avoided). For ARM the position could be 0x8000, the standard ET_EXEC load address, but that is needlessly close to the NULL address, and anyone running PIE on 32-bit ARM will have an MMU, so the tight mapping is not needed. Link: http://lkml.kernel.org/r/1498154792-49952-2-git-send-email-keescook@chromium.org Signed-off-by: Kees Cook <keescook@chromium.org> Cc: Russell King <linux@armlinux.org.uk> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will.deacon@arm.com> Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org> Cc: Paul Mackerras <paulus@samba.org> Cc: Michael Ellerman <mpe@ellerman.id.au> Cc: Martin Schwidefsky <schwidefsky@de.ibm.com> Cc: Heiko Carstens <heiko.carstens@de.ibm.com> Cc: James Hogan <james.hogan@imgtec.com> Cc: Pratyush Anand <panand@redhat.com> Cc: Ingo Molnar <mingo@kernel.org> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Alexander Viro <viro@zeniv.linux.org.uk> Cc: Andy Lutomirski <luto@amacapital.net> Cc: Daniel Micay <danielmicay@gmail.com> Cc: Dmitry Safonov <dsafonov@virtuozzo.com> Cc: Grzegorz Andrejczuk <grzegorz.andrejczuk@intel.com> Cc: Kees Cook <keescook@chromium.org> Cc: Masahiro Yamada <yamada.masahiro@socionext.com> Cc: Qualys Security Advisory <qsa@qualys.com> Cc: Rik van Riel <riel@redhat.com> Cc: Thomas Gleixner <tglx@linutronix.de> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org> diff --git a/arch/arm/include/asm/elf.h b/arch/arm/include/asm/elf.h index d2315ff..f13ae15 100644 --- a/arch/arm/include/asm/elf.h +++ b/arch/arm/include/asm/elf.h @@ -112,12 +112,8 @@ int dump_task_regs(struct task_struct *t, elf_gregset_t *elfregs); #define CORE_DUMP_USE_REGSET #define ELF_EXEC_PAGESIZE 4096 -/* This is the location that an ET_DYN program is loaded if exec'ed. Typical - use of this is to invoke "./ld.so someprog" to test out a new version of - the loader. We need to make sure that it is out of the way of the program - that it will "exec", and that there is sufficient room for the brk. */ - -#define ELF_ET_DYN_BASE (TASK_SIZE / 3 * 2) +/* This is the base location for PIE (ET_DYN with INTERP) loads. */ +#define ELF_ET_DYN_BASE 0x400000UL /* When the program starts, a1 contains a pointer to a function to be registered with atexit, as per the SVR4 ABI. A value of 0 means we - kernel patch分析
-
load_elf_binary()关键代码如下:
static int load_elf_binary(struct linux_binprm *bprm) { ... ... if (elf_interpreter) { load_bias = ELF_ET_DYN_BASE; if (current->flags & PF_RANDOMIZE) load_bias += arch_mmap_rnd(); elf_flags |= MAP_FIXED; } else load_bias = 0; /* * Since load_bias is used for all subsequent loading * calculations, we must lower it by the first vaddr * so that the remaining calculations based on the * ELF vaddrs will be correctly offset. The result * is then page aligned. */ load_bias = ELF_PAGESTART(load_bias - vaddr); total_size = total_mapping_size(elf_phdata, loc->elf_ex.e_phnum); if (!total_size) { retval = -EINVAL; goto out_free_dentry; } } error = elf_map(bprm->file, load_bias + vaddr, elf_ppnt, elf_prot, elf_flags, total_size); ... ... -
arch_mmap_rnd()
unsigned long arch_mmap_rnd(void) { unsigned long rnd; rnd = get_random_long() & ((1UL << mmap_rnd_bits) - 1); return rnd << PAGE_SHIFT; }这里先调用get_random_long()获取一个随机数,假设为0xXXXXXXXX,
(0xXXXXXXXX & ((1 << 16) - 1)) << 12 = 0x0XXXX000，最后返回的结果不会超过0x0ffff000
-
patch引入之前
ELF_ET_DYN_BASE = (TASK_SIZE / 3 * 2) = (0xC0000000 - 0x01000000) / 3 * 2 = 0x7f555554
load_bias = 0x7f555554
load_bias += random(不超过0x0ffff000)
load_bias最后再和vaddr做page对齐
-
所以在32bit arm上看到可执行程序经常映射到
0x8XXXXXXX的． 如:8cd23000-8cd42000 r-xp 00000000 103:0b 287 /system/bin/dex2oat 8cd43000-8cd44000 r--p 0001f000 103:0b 287 /system/bin/dex2oat 8d268000-8d26d000 r-xp 00000000 103:0b 231 /system/bin/app_process32 8d26d000-8d26e000 r--p 00004000 103:0b 231 /system/bin/app_process32
-
patch引入之后
ELF_ET_DYN_BASE = 0x400000
load_bias = 0x400000
load_bias += random(不超过0x0ffff000)
load_bias最后再和vaddr做page对齐
-
所以在32bit arm上看到可执行程序经常映射到
0x10000000以内． 如:0fa73000-0fa91fff r-x 0 1f000 /system/bin/dex2oat (BuildId: 98d4fb6e838289e7a2bc42b4caf1d455) 0fa93000-0fa93fff r-- 1f000 1000 /system/bin/dex2oat 0564c000-05651000 r-xp 00000000 103:0b 230 /system/bin/app_process32 05651000-05652000 r--p 00004000 103:0b 230 /system/bin/app_process32 05652000-05653000 rw-p 00000000 00:00 0 所以，这里就容易和ART默认要mmap到的0x0ec00000-0x12c00000产生冲突.
-
Root Cause
- 该问题本质上是kernel与art关于虚拟地址空间的映射范围产生潜在的冲突．
解决方案
-
已将此问题反馈给Google的Vladimir Marko(vmarko@google.com)
回复如下:
I guess we may need to rethink the address range.
-
patch
目前先临时回退kernel的相关提交，后续问题得到明确修复后再修改．
-
验证
基于回退后的版本，可执行程序已能映射到高地址．
