tjh.dev / kernel
Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

Merge branch 'emit-endbr-bti-instructions-for-indirect'

Xu Kuohai says:

====================
emit ENDBR/BTI instructions for indirect

On architectures with CFI protection enabled that require landing pad
instructions at indirect jump targets, such as x86 with CET/IBT enabled
and arm64 with BTI enabled, kernel panics when an indirect jump lands on
a target without landing pad. Therefore, the JIT must emit landing pad
instructions for indirect jump targets.

The verifier already recognizes which instructions are indirect jump
targets during the verification phase. So we can store this information
in env->insn_aux_data and pass it to the JIT as new parameter, allowing
the JIT to consult env->insn_aux_data to determine which instructions are
indirect jump targets.

During JIT, constants blinding is performed. It rewrites the private copy
of instructions for the JITed program, but it does not adjust the global
env->insn_aux_data array. As a result, after constants blinding, the
instruction indexes used by JIT may no longer match the indexes in
env->insn_aux_data, so the JIT can not use env->insn_aux_data directly.

To avoid this mismatch, and given that all existing arch-specific JITs
already implement constants blinding with largely duplicated code, move
constants blinding from JIT to generic code.

v15:
- Rebase and target bpf tree
- Resotre subprog_start of the fake 'exit' subprog on failure
- Fix wrong function name used in comment

v14: https://lore.kernel.org/all/cover.1776062885.git.xukuohai@hotmail.com/
- Rebase
- Fix comment style
- Fix incorrect variable and function name used in commit message

v13: https://lore.kernel.org/bpf/20260411133847.1042658-1-xukuohai@huaweicloud.com
- Use vmalloc to allocate memory for insn_aux_data copies to match with vfree
- Do not free the copied memory of insn_aux_data when restoring from failure
- Code cleanup

v12: https://lore.kernel.org/bpf/20260403132811.753894-1-xukuohai@huaweicloud.com
- Restore env->insn_aux_data on JIT failure
- Fix incorrect error code sign (-EFAULT vs EFAULT)
- Fix incorrect prog used in the restore path

v11: https://lore.kernel.org/bpf/20260403090915.473493-1-xukuohai@huaweicloud.com
- Restore env->subprog_info after jit_subprogs() fails
- Clear prog->jit_requested and prog->blinding_requested on failure
- Use the actual env->insn_aux_data size in clear_insn_aux_data() on failure

v10: https://lore.kernel.org/bpf/20260324122052.342751-1-xukuohai@huaweicloud.com
- Fix the incorrect call_imm restore in jit_subprogs
- Define a dummy void version of bpf_jit_prog_release_other and
bpf_patch_insn_data when the corresponding config is not set
- Remove the unnecessary #ifdef in x86_64 JIT (Leon Hwang)

v9: https://lore.kernel.org/bpf/20260312170255.3427799-1-xukuohai@huaweicloud.com
- Make constant blinding available for classic bpf (Eduard)
- Clear prog->bpf_func, prog->jited ... on the error path of extra pass (Eduard)
- Fix spelling errors and remove unused parameter (Anton Protopopov)

v8: https://lore.kernel.org/bpf/20260309140044.2652538-1-xukuohai@huaweicloud.com
- Define void bpf_jit_blind_constants() function when CONFIG_BPF_JIT is not set
- Move indirect_target fixup for insn patching from bpf_jit_blind_constants()
to adjust_insn_aux_data()

v7: https://lore.kernel.org/bpf/20260307103949.2340104-1-xukuohai@huaweicloud.com
- Move constants blinding logic back to bpf/core.c
- Compute ip address before switch statement in x86 JIT
- Clear JIT state from error path on arm64 and loongarch

v6: https://lore.kernel.org/bpf/20260306102329.2056216-1-xukuohai@huaweicloud.com
- Move constants blinding from JIT to verifier
- Move call to bpf_prog_select_runtime from bpf_prog_load to verifier

v5: https://lore.kernel.org/bpf/20260302102726.1126019-1-xukuohai@huaweicloud.com
- Switch to pass env to JIT directly to get rid of copying private insn_aux_data for
each prog

v4: https://lore.kernel.org/all/20260114093914.2403982-1-xukuohai@huaweicloud.com
- Switch to the approach proposed by Eduard, using insn_aux_data to identify indirect
jump targets, and emit ENDBR on x86

v3: https://lore.kernel.org/bpf/20251227081033.240336-1-xukuohai@huaweicloud.com
- Get rid of unnecessary enum definition (Yonghong Song, Anton Protopopov)

v2: https://lore.kernel.org/bpf/20251223085447.139301-1-xukuohai@huaweicloud.com
- Exclude instruction arrays not used for indirect jumps (Anton Protopopov)

v1: https://lore.kernel.org/bpf/20251127140318.3944249-1-xukuohai@huaweicloud.com
====================

Link: https://patch.msgid.link/20260416064341.151802-1-xukuohai@huaweicloud.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>

Alexei Starovoitov 1cedfe17 a2044665

+529 -555
+15 -26
arch/arc/net/bpf_jit_core.c
··· 79 79 * The JIT pertinent context that is used by different functions. 80 80 * 81 81 * prog: The current eBPF program being handled. 82 - * orig_prog: The original eBPF program before any possible change. 83 82 * jit: The JIT buffer and its length. 84 83 * bpf_header: The JITed program header. "jit.buf" points inside it. 85 84 * emit: If set, opcodes are written to memory; else, a dry-run. ··· 93 94 * need_extra_pass: A forecast if an "extra_pass" will occur. 94 95 * is_extra_pass: Indicates if the current pass is an extra pass. 95 96 * user_bpf_prog: True, if VM opcodes come from a real program. 96 - * blinded: True if "constant blinding" step returned a new "prog". 97 97 * success: Indicates if the whole JIT went OK. 98 98 */ 99 99 struct jit_context { 100 100 struct bpf_prog *prog; 101 - struct bpf_prog *orig_prog; 102 101 struct jit_buffer jit; 103 102 struct bpf_binary_header *bpf_header; 104 103 bool emit; ··· 111 114 bool need_extra_pass; 112 115 bool is_extra_pass; 113 116 bool user_bpf_prog; 114 - bool blinded; 115 117 bool success; 116 118 }; 117 119 ··· 157 161 { 158 162 memset(ctx, 0, sizeof(*ctx)); 159 163 160 - ctx->orig_prog = prog; 161 - 162 - /* If constant blinding was requested but failed, scram. */ 163 - ctx->prog = bpf_jit_blind_constants(prog); 164 - if (IS_ERR(ctx->prog)) 165 - return PTR_ERR(ctx->prog); 166 - ctx->blinded = (ctx->prog != ctx->orig_prog); 164 + ctx->prog = prog; 167 165 168 166 /* If the verifier doesn't zero-extend, then we have to do it. */ 169 167 ctx->do_zext = !ctx->prog->aux->verifier_zext; ··· 204 214 */ 205 215 static void jit_ctx_cleanup(struct jit_context *ctx) 206 216 { 207 - if (ctx->blinded) { 208 - /* if all went well, release the orig_prog. */ 209 - if (ctx->success) 210 - bpf_jit_prog_release_other(ctx->prog, ctx->orig_prog); 211 - else 212 - bpf_jit_prog_release_other(ctx->orig_prog, ctx->prog); 213 - } 214 - 215 217 maybe_free(ctx, (void **)&ctx->bpf2insn); 216 218 maybe_free(ctx, (void **)&ctx->jit_data); 217 219 ··· 211 229 ctx->bpf2insn_valid = false; 212 230 213 231 /* Freeing "bpf_header" is enough. "jit.buf" is a sub-array of it. */ 214 - if (!ctx->success && ctx->bpf_header) { 215 - bpf_jit_binary_free(ctx->bpf_header); 216 - ctx->bpf_header = NULL; 217 - ctx->jit.buf = NULL; 218 - ctx->jit.index = 0; 219 - ctx->jit.len = 0; 232 + if (!ctx->success) { 233 + if (ctx->bpf_header) { 234 + bpf_jit_binary_free(ctx->bpf_header); 235 + ctx->bpf_header = NULL; 236 + ctx->jit.buf = NULL; 237 + ctx->jit.index = 0; 238 + ctx->jit.len = 0; 239 + } 240 + if (ctx->is_extra_pass) { 241 + ctx->prog->bpf_func = NULL; 242 + ctx->prog->jited = 0; 243 + ctx->prog->jited_len = 0; 244 + } 220 245 } 221 246 222 247 ctx->emit = false; ··· 1400 1411 * (re)locations involved that their addresses are not known 1401 1412 * during the first run. 1402 1413 */ 1403 - struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 1414 + struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog) 1404 1415 { 1405 1416 vm_dump(prog); 1406 1417
+8 -35
arch/arm/net/bpf_jit_32.c
··· 2142 2142 return true; 2143 2143 } 2144 2144 2145 - struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 2145 + struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog) 2146 2146 { 2147 - struct bpf_prog *tmp, *orig_prog = prog; 2148 2147 struct bpf_binary_header *header; 2149 - bool tmp_blinded = false; 2150 2148 struct jit_ctx ctx; 2151 2149 unsigned int tmp_idx; 2152 2150 unsigned int image_size; ··· 2154 2156 * the interpreter. 2155 2157 */ 2156 2158 if (!prog->jit_requested) 2157 - return orig_prog; 2158 - 2159 - /* If constant blinding was enabled and we failed during blinding 2160 - * then we must fall back to the interpreter. Otherwise, we save 2161 - * the new JITed code. 2162 - */ 2163 - tmp = bpf_jit_blind_constants(prog); 2164 - 2165 - if (IS_ERR(tmp)) 2166 - return orig_prog; 2167 - if (tmp != prog) { 2168 - tmp_blinded = true; 2169 - prog = tmp; 2170 - } 2159 + return prog; 2171 2160 2172 2161 memset(&ctx, 0, sizeof(ctx)); 2173 2162 ctx.prog = prog; ··· 2164 2179 * we must fall back to the interpreter 2165 2180 */ 2166 2181 ctx.offsets = kcalloc(prog->len, sizeof(int), GFP_KERNEL); 2167 - if (ctx.offsets == NULL) { 2168 - prog = orig_prog; 2169 - goto out; 2170 - } 2182 + if (ctx.offsets == NULL) 2183 + return prog; 2171 2184 2172 2185 /* 1) fake pass to find in the length of the JITed code, 2173 2186 * to compute ctx->offsets and other context variables ··· 2177 2194 * being successful in the second pass, so just fall back 2178 2195 * to the interpreter. 2179 2196 */ 2180 - if (build_body(&ctx)) { 2181 - prog = orig_prog; 2197 + if (build_body(&ctx)) 2182 2198 goto out_off; 2183 - } 2184 2199 2185 2200 tmp_idx = ctx.idx; 2186 2201 build_prologue(&ctx); ··· 2194 2213 ctx.idx += ctx.imm_count; 2195 2214 if (ctx.imm_count) { 2196 2215 ctx.imms = kcalloc(ctx.imm_count, sizeof(u32), GFP_KERNEL); 2197 - if (ctx.imms == NULL) { 2198 - prog = orig_prog; 2216 + if (ctx.imms == NULL) 2199 2217 goto out_off; 2200 - } 2201 2218 } 2202 2219 #else 2203 2220 /* there's nothing about the epilogue on ARMv7 */ ··· 2217 2238 /* Not able to allocate memory for the structure then 2218 2239 * we must fall back to the interpretation 2219 2240 */ 2220 - if (header == NULL) { 2221 - prog = orig_prog; 2241 + if (header == NULL) 2222 2242 goto out_imms; 2223 - } 2224 2243 2225 2244 /* 2.) Actual pass to generate final JIT code */ 2226 2245 ctx.target = (u32 *) image_ptr; ··· 2255 2278 #endif 2256 2279 out_off: 2257 2280 kfree(ctx.offsets); 2258 - out: 2259 - if (tmp_blinded) 2260 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 2261 - tmp : orig_prog); 2281 + 2262 2282 return prog; 2263 2283 2264 2284 out_free: 2265 2285 image_ptr = NULL; 2266 2286 bpf_jit_binary_free(header); 2267 - prog = orig_prog; 2268 2287 goto out_imms; 2269 2288 } 2270 2289
+29 -58
arch/arm64/net/bpf_jit_comp.c
··· 1197 1197 * >0 - successfully JITed a 16-byte eBPF instruction. 1198 1198 * <0 - failed to JIT. 1199 1199 */ 1200 - static int build_insn(const struct bpf_insn *insn, struct jit_ctx *ctx, 1201 - bool extra_pass) 1200 + static int build_insn(const struct bpf_verifier_env *env, const struct bpf_insn *insn, 1201 + struct jit_ctx *ctx, bool extra_pass) 1202 1202 { 1203 1203 const u8 code = insn->code; 1204 1204 u8 dst = bpf2a64[insn->dst_reg]; ··· 1222 1222 int off_adj; 1223 1223 int ret; 1224 1224 bool sign_extend; 1225 + 1226 + if (bpf_insn_is_indirect_target(env, ctx->prog, i)) 1227 + emit_bti(A64_BTI_J, ctx); 1225 1228 1226 1229 switch (code) { 1227 1230 /* dst = src */ ··· 1901 1898 return 0; 1902 1899 } 1903 1900 1904 - static int build_body(struct jit_ctx *ctx, bool extra_pass) 1901 + static int build_body(struct bpf_verifier_env *env, struct jit_ctx *ctx, bool extra_pass) 1905 1902 { 1906 1903 const struct bpf_prog *prog = ctx->prog; 1907 1904 int i; ··· 1920 1917 int ret; 1921 1918 1922 1919 ctx->offset[i] = ctx->idx; 1923 - ret = build_insn(insn, ctx, extra_pass); 1920 + ret = build_insn(env, insn, ctx, extra_pass); 1924 1921 if (ret > 0) { 1925 1922 i++; 1926 1923 ctx->offset[i] = ctx->idx; ··· 2003 2000 struct jit_ctx ctx; 2004 2001 }; 2005 2002 2006 - struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 2003 + struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog) 2007 2004 { 2008 2005 int image_size, prog_size, extable_size, extable_align, extable_offset; 2009 - struct bpf_prog *tmp, *orig_prog = prog; 2010 2006 struct bpf_binary_header *header; 2011 2007 struct bpf_binary_header *ro_header = NULL; 2012 2008 struct arm64_jit_data *jit_data; 2013 2009 void __percpu *priv_stack_ptr = NULL; 2014 2010 bool was_classic = bpf_prog_was_classic(prog); 2015 2011 int priv_stack_alloc_sz; 2016 - bool tmp_blinded = false; 2017 2012 bool extra_pass = false; 2018 2013 struct jit_ctx ctx; 2019 2014 u8 *image_ptr; ··· 2020 2019 int exentry_idx; 2021 2020 2022 2021 if (!prog->jit_requested) 2023 - return orig_prog; 2024 - 2025 - tmp = bpf_jit_blind_constants(prog); 2026 - /* If blinding was requested and we failed during blinding, 2027 - * we must fall back to the interpreter. 2028 - */ 2029 - if (IS_ERR(tmp)) 2030 - return orig_prog; 2031 - if (tmp != prog) { 2032 - tmp_blinded = true; 2033 - prog = tmp; 2034 - } 2022 + return prog; 2035 2023 2036 2024 jit_data = prog->aux->jit_data; 2037 2025 if (!jit_data) { 2038 2026 jit_data = kzalloc_obj(*jit_data); 2039 - if (!jit_data) { 2040 - prog = orig_prog; 2041 - goto out; 2042 - } 2027 + if (!jit_data) 2028 + return prog; 2043 2029 prog->aux->jit_data = jit_data; 2044 2030 } 2045 2031 priv_stack_ptr = prog->aux->priv_stack_ptr; ··· 2038 2050 priv_stack_alloc_sz = round_up(prog->aux->stack_depth, 16) + 2039 2051 2 * PRIV_STACK_GUARD_SZ; 2040 2052 priv_stack_ptr = __alloc_percpu_gfp(priv_stack_alloc_sz, 16, GFP_KERNEL); 2041 - if (!priv_stack_ptr) { 2042 - prog = orig_prog; 2053 + if (!priv_stack_ptr) 2043 2054 goto out_priv_stack; 2044 - } 2045 2055 2046 2056 priv_stack_init_guard(priv_stack_ptr, priv_stack_alloc_sz); 2047 2057 prog->aux->priv_stack_ptr = priv_stack_ptr; ··· 2059 2073 ctx.prog = prog; 2060 2074 2061 2075 ctx.offset = kvzalloc_objs(int, prog->len + 1); 2062 - if (ctx.offset == NULL) { 2063 - prog = orig_prog; 2076 + if (ctx.offset == NULL) 2064 2077 goto out_off; 2065 - } 2066 2078 2067 2079 ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena); 2068 2080 ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena); ··· 2073 2089 * BPF line info needs ctx->offset[i] to be the offset of 2074 2090 * instruction[i] in jited image, so build prologue first. 2075 2091 */ 2076 - if (build_prologue(&ctx, was_classic)) { 2077 - prog = orig_prog; 2092 + if (build_prologue(&ctx, was_classic)) 2078 2093 goto out_off; 2079 - } 2080 2094 2081 - if (build_body(&ctx, extra_pass)) { 2082 - prog = orig_prog; 2095 + if (build_body(env, &ctx, extra_pass)) 2083 2096 goto out_off; 2084 - } 2085 2097 2086 2098 ctx.epilogue_offset = ctx.idx; 2087 2099 build_epilogue(&ctx, was_classic); ··· 2095 2115 ro_header = bpf_jit_binary_pack_alloc(image_size, &ro_image_ptr, 2096 2116 sizeof(u64), &header, &image_ptr, 2097 2117 jit_fill_hole); 2098 - if (!ro_header) { 2099 - prog = orig_prog; 2118 + if (!ro_header) 2100 2119 goto out_off; 2101 - } 2102 2120 2103 2121 /* Pass 2: Determine jited position and result for each instruction */ 2104 2122 ··· 2124 2146 /* Dont write body instructions to memory for now */ 2125 2147 ctx.write = false; 2126 2148 2127 - if (build_body(&ctx, extra_pass)) { 2128 - prog = orig_prog; 2149 + if (build_body(env, &ctx, extra_pass)) 2129 2150 goto out_free_hdr; 2130 - } 2131 2151 2132 2152 ctx.epilogue_offset = ctx.idx; 2133 2153 ctx.exentry_idx = exentry_idx; ··· 2133 2157 ctx.write = true; 2134 2158 2135 2159 /* Pass 3: Adjust jump offset and write final image */ 2136 - if (build_body(&ctx, extra_pass) || 2137 - WARN_ON_ONCE(ctx.idx != ctx.epilogue_offset)) { 2138 - prog = orig_prog; 2160 + if (build_body(env, &ctx, extra_pass) || 2161 + WARN_ON_ONCE(ctx.idx != ctx.epilogue_offset)) 2139 2162 goto out_free_hdr; 2140 - } 2141 2163 2142 2164 build_epilogue(&ctx, was_classic); 2143 2165 build_plt(&ctx); 2144 2166 2145 2167 /* Extra pass to validate JITed code. */ 2146 - if (validate_ctx(&ctx)) { 2147 - prog = orig_prog; 2168 + if (validate_ctx(&ctx)) 2148 2169 goto out_free_hdr; 2149 - } 2150 2170 2151 2171 /* update the real prog size */ 2152 2172 prog_size = sizeof(u32) * ctx.idx; ··· 2159 2187 if (extra_pass && ctx.idx > jit_data->ctx.idx) { 2160 2188 pr_err_once("multi-func JIT bug %d > %d\n", 2161 2189 ctx.idx, jit_data->ctx.idx); 2162 - prog->bpf_func = NULL; 2163 - prog->jited = 0; 2164 - prog->jited_len = 0; 2165 2190 goto out_free_hdr; 2166 2191 } 2167 2192 if (WARN_ON(bpf_jit_binary_pack_finalize(ro_header, header))) { 2168 - /* ro_header has been freed */ 2193 + /* ro_header and header has been freed */ 2169 2194 ro_header = NULL; 2170 - prog = orig_prog; 2171 - goto out_off; 2195 + header = NULL; 2196 + goto out_free_hdr; 2172 2197 } 2173 2198 } else { 2174 2199 jit_data->ctx = ctx; ··· 2202 2233 kfree(jit_data); 2203 2234 prog->aux->jit_data = NULL; 2204 2235 } 2205 - out: 2206 - if (tmp_blinded) 2207 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 2208 - tmp : orig_prog); 2236 + 2209 2237 return prog; 2210 2238 2211 2239 out_free_hdr: 2240 + if (extra_pass) { 2241 + prog->bpf_func = NULL; 2242 + prog->jited = 0; 2243 + prog->jited_len = 0; 2244 + } 2212 2245 if (header) { 2213 2246 bpf_arch_text_copy(&ro_header->size, &header->size, 2214 2247 sizeof(header->size));
+18 -43
arch/loongarch/net/bpf_jit.c
··· 1920 1920 return ret < 0 ? ret : ret * LOONGARCH_INSN_SIZE; 1921 1921 } 1922 1922 1923 - struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 1923 + struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog) 1924 1924 { 1925 - bool tmp_blinded = false, extra_pass = false; 1925 + bool extra_pass = false; 1926 1926 u8 *image_ptr, *ro_image_ptr; 1927 1927 int image_size, prog_size, extable_size; 1928 1928 struct jit_ctx ctx; 1929 1929 struct jit_data *jit_data; 1930 1930 struct bpf_binary_header *header; 1931 1931 struct bpf_binary_header *ro_header; 1932 - struct bpf_prog *tmp, *orig_prog = prog; 1933 1932 1934 1933 /* 1935 1934 * If BPF JIT was not enabled then we must fall back to 1936 1935 * the interpreter. 1937 1936 */ 1938 1937 if (!prog->jit_requested) 1939 - return orig_prog; 1940 - 1941 - tmp = bpf_jit_blind_constants(prog); 1942 - /* 1943 - * If blinding was requested and we failed during blinding, 1944 - * we must fall back to the interpreter. Otherwise, we save 1945 - * the new JITed code. 1946 - */ 1947 - if (IS_ERR(tmp)) 1948 - return orig_prog; 1949 - 1950 - if (tmp != prog) { 1951 - tmp_blinded = true; 1952 - prog = tmp; 1953 - } 1938 + return prog; 1954 1939 1955 1940 jit_data = prog->aux->jit_data; 1956 1941 if (!jit_data) { 1957 1942 jit_data = kzalloc_obj(*jit_data); 1958 - if (!jit_data) { 1959 - prog = orig_prog; 1960 - goto out; 1961 - } 1943 + if (!jit_data) 1944 + return prog; 1962 1945 prog->aux->jit_data = jit_data; 1963 1946 } 1964 1947 if (jit_data->ctx.offset) { ··· 1961 1978 ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena); 1962 1979 1963 1980 ctx.offset = kvcalloc(prog->len + 1, sizeof(u32), GFP_KERNEL); 1964 - if (ctx.offset == NULL) { 1965 - prog = orig_prog; 1981 + if (ctx.offset == NULL) 1966 1982 goto out_offset; 1967 - } 1968 1983 1969 1984 /* 1. Initial fake pass to compute ctx->idx and set ctx->flags */ 1970 1985 build_prologue(&ctx); 1971 - if (build_body(&ctx, extra_pass)) { 1972 - prog = orig_prog; 1986 + if (build_body(&ctx, extra_pass)) 1973 1987 goto out_offset; 1974 - } 1975 1988 ctx.epilogue_offset = ctx.idx; 1976 1989 build_epilogue(&ctx); 1977 1990 ··· 1983 2004 /* Now we know the size of the structure to make */ 1984 2005 ro_header = bpf_jit_binary_pack_alloc(image_size, &ro_image_ptr, sizeof(u32), 1985 2006 &header, &image_ptr, jit_fill_hole); 1986 - if (!ro_header) { 1987 - prog = orig_prog; 2007 + if (!ro_header) 1988 2008 goto out_offset; 1989 - } 1990 2009 1991 2010 /* 2. Now, the actual pass to generate final JIT code */ 1992 2011 /* ··· 2004 2027 ctx.num_exentries = 0; 2005 2028 2006 2029 build_prologue(&ctx); 2007 - if (build_body(&ctx, extra_pass)) { 2008 - prog = orig_prog; 2030 + if (build_body(&ctx, extra_pass)) 2009 2031 goto out_free; 2010 - } 2011 2032 build_epilogue(&ctx); 2012 2033 2013 2034 /* 3. Extra pass to validate JITed code */ 2014 - if (validate_ctx(&ctx)) { 2015 - prog = orig_prog; 2035 + if (validate_ctx(&ctx)) 2016 2036 goto out_free; 2017 - } 2018 2037 2019 2038 /* And we're done */ 2020 2039 if (bpf_jit_enable > 1) ··· 2023 2050 goto out_free; 2024 2051 } 2025 2052 if (WARN_ON(bpf_jit_binary_pack_finalize(ro_header, header))) { 2026 - /* ro_header has been freed */ 2053 + /* ro_header and header have been freed */ 2027 2054 ro_header = NULL; 2028 - prog = orig_prog; 2055 + header = NULL; 2029 2056 goto out_free; 2030 2057 } 2031 2058 /* ··· 2057 2084 prog->aux->jit_data = NULL; 2058 2085 } 2059 2086 2060 - out: 2061 - if (tmp_blinded) 2062 - bpf_jit_prog_release_other(prog, prog == orig_prog ? tmp : orig_prog); 2063 - 2064 2087 return prog; 2065 2088 2066 2089 out_free: 2090 + if (extra_pass) { 2091 + prog->bpf_func = NULL; 2092 + prog->jited = 0; 2093 + prog->jited_len = 0; 2094 + } 2095 + 2067 2096 if (header) { 2068 2097 bpf_arch_text_copy(&ro_header->size, &header->size, sizeof(header->size)); 2069 2098 bpf_jit_binary_pack_free(ro_header, header);
+2 -20
arch/mips/net/bpf_jit_comp.c
··· 909 909 return true; 910 910 } 911 911 912 - struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 912 + struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog) 913 913 { 914 - struct bpf_prog *tmp, *orig_prog = prog; 915 914 struct bpf_binary_header *header = NULL; 916 915 struct jit_context ctx; 917 - bool tmp_blinded = false; 918 916 unsigned int tmp_idx; 919 917 unsigned int image_size; 920 918 u8 *image_ptr; ··· 923 925 * the interpreter. 924 926 */ 925 927 if (!prog->jit_requested) 926 - return orig_prog; 927 - /* 928 - * If constant blinding was enabled and we failed during blinding 929 - * then we must fall back to the interpreter. Otherwise, we save 930 - * the new JITed code. 931 - */ 932 - tmp = bpf_jit_blind_constants(prog); 933 - if (IS_ERR(tmp)) 934 - return orig_prog; 935 - if (tmp != prog) { 936 - tmp_blinded = true; 937 - prog = tmp; 938 - } 928 + return prog; 939 929 940 930 memset(&ctx, 0, sizeof(ctx)); 941 931 ctx.program = prog; ··· 1011 1025 prog->jited_len = image_size; 1012 1026 1013 1027 out: 1014 - if (tmp_blinded) 1015 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 1016 - tmp : orig_prog); 1017 1028 kfree(ctx.descriptors); 1018 1029 return prog; 1019 1030 1020 1031 out_err: 1021 - prog = orig_prog; 1022 1032 if (header) 1023 1033 bpf_jit_binary_free(header); 1024 1034 goto out;
+30 -45
arch/parisc/net/bpf_jit_core.c
··· 41 41 return true; 42 42 } 43 43 44 - struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 44 + struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog) 45 45 { 46 46 unsigned int prog_size = 0, extable_size = 0; 47 - bool tmp_blinded = false, extra_pass = false; 48 - struct bpf_prog *tmp, *orig_prog = prog; 47 + bool extra_pass = false; 49 48 int pass = 0, prev_ninsns = 0, prologue_len, i; 50 49 struct hppa_jit_data *jit_data; 51 50 struct hppa_jit_context *ctx; 52 51 53 52 if (!prog->jit_requested) 54 - return orig_prog; 55 - 56 - tmp = bpf_jit_blind_constants(prog); 57 - if (IS_ERR(tmp)) 58 - return orig_prog; 59 - if (tmp != prog) { 60 - tmp_blinded = true; 61 - prog = tmp; 62 - } 53 + return prog; 63 54 64 55 jit_data = prog->aux->jit_data; 65 56 if (!jit_data) { 66 57 jit_data = kzalloc_obj(*jit_data); 67 - if (!jit_data) { 68 - prog = orig_prog; 69 - goto out; 70 - } 58 + if (!jit_data) 59 + return prog; 71 60 prog->aux->jit_data = jit_data; 72 61 } 73 62 ··· 70 81 71 82 ctx->prog = prog; 72 83 ctx->offset = kzalloc_objs(int, prog->len); 73 - if (!ctx->offset) { 74 - prog = orig_prog; 75 - goto out_offset; 76 - } 84 + if (!ctx->offset) 85 + goto out_err; 77 86 for (i = 0; i < prog->len; i++) { 78 87 prev_ninsns += 20; 79 88 ctx->offset[i] = prev_ninsns; ··· 80 93 for (i = 0; i < NR_JIT_ITERATIONS; i++) { 81 94 pass++; 82 95 ctx->ninsns = 0; 83 - if (build_body(ctx, extra_pass, ctx->offset)) { 84 - prog = orig_prog; 85 - goto out_offset; 86 - } 96 + if (build_body(ctx, extra_pass, ctx->offset)) 97 + goto out_err; 87 98 ctx->body_len = ctx->ninsns; 88 99 bpf_jit_build_prologue(ctx); 89 100 ctx->prologue_len = ctx->ninsns - ctx->body_len; ··· 101 116 &jit_data->image, 102 117 sizeof(long), 103 118 bpf_fill_ill_insns); 104 - if (!jit_data->header) { 105 - prog = orig_prog; 106 - goto out_offset; 107 - } 119 + if (!jit_data->header) 120 + goto out_err; 108 121 109 122 ctx->insns = (u32 *)jit_data->image; 110 123 /* ··· 117 134 pr_err("bpf-jit: image did not converge in <%d passes!\n", i); 118 135 if (jit_data->header) 119 136 bpf_jit_binary_free(jit_data->header); 120 - prog = orig_prog; 121 - goto out_offset; 137 + goto out_err; 122 138 } 123 139 124 140 if (extable_size) ··· 130 148 bpf_jit_build_prologue(ctx); 131 149 if (build_body(ctx, extra_pass, NULL)) { 132 150 bpf_jit_binary_free(jit_data->header); 133 - prog = orig_prog; 134 - goto out_offset; 151 + goto out_err; 135 152 } 136 153 bpf_jit_build_epilogue(ctx); 137 154 ··· 141 160 { extern int machine_restart(char *); machine_restart(""); } 142 161 } 143 162 163 + if (!prog->is_func || extra_pass) { 164 + if (bpf_jit_binary_lock_ro(jit_data->header)) { 165 + bpf_jit_binary_free(jit_data->header); 166 + goto out_err; 167 + } 168 + bpf_flush_icache(jit_data->header, ctx->insns + ctx->ninsns); 169 + } 170 + 144 171 prog->bpf_func = (void *)ctx->insns; 145 172 prog->jited = 1; 146 173 prog->jited_len = prog_size; 147 174 148 - bpf_flush_icache(jit_data->header, ctx->insns + ctx->ninsns); 149 - 150 175 if (!prog->is_func || extra_pass) { 151 - if (bpf_jit_binary_lock_ro(jit_data->header)) { 152 - bpf_jit_binary_free(jit_data->header); 153 - prog->bpf_func = NULL; 154 - prog->jited = 0; 155 - prog->jited_len = 0; 156 - goto out_offset; 157 - } 158 176 prologue_len = ctx->epilogue_offset - ctx->body_len; 159 177 for (i = 0; i < prog->len; i++) 160 178 ctx->offset[i] += prologue_len; ··· 163 183 kfree(jit_data); 164 184 prog->aux->jit_data = NULL; 165 185 } 166 - out: 186 + 167 187 if (HPPA_JIT_REBOOT) 168 188 { extern int machine_restart(char *); machine_restart(""); } 169 189 170 - if (tmp_blinded) 171 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 172 - tmp : orig_prog); 173 190 return prog; 191 + 192 + out_err: 193 + if (extra_pass) { 194 + prog->bpf_func = NULL; 195 + prog->jited = 0; 196 + prog->jited_len = 0; 197 + } 198 + goto out_offset; 174 199 } 175 200 176 201 u64 hppa_div64(u64 div, u64 divisor)
+27 -47
arch/powerpc/net/bpf_jit_comp.c
··· 162 162 } 163 163 } 164 164 165 - struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) 165 + struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *fp) 166 166 { 167 167 u32 proglen; 168 168 u32 alloclen; ··· 177 177 void __percpu *priv_stack_ptr = NULL; 178 178 struct bpf_binary_header *fhdr = NULL; 179 179 struct bpf_binary_header *hdr = NULL; 180 - struct bpf_prog *org_fp = fp; 181 - struct bpf_prog *tmp_fp = NULL; 182 - bool bpf_blinded = false; 183 180 bool extra_pass = false; 184 181 u8 *fimage = NULL; 185 182 u32 *fcode_base = NULL; ··· 184 187 u32 fixup_len; 185 188 186 189 if (!fp->jit_requested) 187 - return org_fp; 188 - 189 - tmp_fp = bpf_jit_blind_constants(org_fp); 190 - if (IS_ERR(tmp_fp)) 191 - return org_fp; 192 - 193 - if (tmp_fp != org_fp) { 194 - bpf_blinded = true; 195 - fp = tmp_fp; 196 - } 190 + return fp; 197 191 198 192 jit_data = fp->aux->jit_data; 199 193 if (!jit_data) { 200 194 jit_data = kzalloc_obj(*jit_data); 201 - if (!jit_data) { 202 - fp = org_fp; 203 - goto out; 204 - } 195 + if (!jit_data) 196 + return fp; 205 197 fp->aux->jit_data = jit_data; 206 198 } 207 199 ··· 205 219 priv_stack_alloc_size = round_up(fp->aux->stack_depth, 16) + 206 220 2 * PRIV_STACK_GUARD_SZ; 207 221 priv_stack_ptr = __alloc_percpu_gfp(priv_stack_alloc_size, 16, GFP_KERNEL); 208 - if (!priv_stack_ptr) { 209 - fp = org_fp; 222 + if (!priv_stack_ptr) 210 223 goto out_priv_stack; 211 - } 212 224 213 225 priv_stack_init_guard(priv_stack_ptr, priv_stack_alloc_size); 214 226 fp->aux->priv_stack_ptr = priv_stack_ptr; ··· 233 249 } 234 250 235 251 addrs = kcalloc(flen + 1, sizeof(*addrs), GFP_KERNEL); 236 - if (addrs == NULL) { 237 - fp = org_fp; 238 - goto out_addrs; 239 - } 252 + if (addrs == NULL) 253 + goto out_err; 240 254 241 255 memset(&cgctx, 0, sizeof(struct codegen_context)); 242 256 bpf_jit_init_reg_mapping(&cgctx); ··· 261 279 } 262 280 263 281 /* Scouting faux-generate pass 0 */ 264 - if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) { 282 + if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) 265 283 /* We hit something illegal or unsupported. */ 266 - fp = org_fp; 267 - goto out_addrs; 268 - } 284 + goto out_err; 269 285 270 286 /* 271 287 * If we have seen a tail call, we need a second pass. ··· 274 294 */ 275 295 if (cgctx.seen & SEEN_TAILCALL || !is_offset_in_branch_range((long)cgctx.idx * 4)) { 276 296 cgctx.idx = 0; 277 - if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) { 278 - fp = org_fp; 279 - goto out_addrs; 280 - } 297 + if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) 298 + goto out_err; 281 299 } 282 300 283 301 bpf_jit_realloc_regs(&cgctx); ··· 296 318 297 319 fhdr = bpf_jit_binary_pack_alloc(alloclen, &fimage, 4, &hdr, &image, 298 320 bpf_jit_fill_ill_insns); 299 - if (!fhdr) { 300 - fp = org_fp; 301 - goto out_addrs; 302 - } 321 + if (!fhdr) 322 + goto out_err; 303 323 304 324 if (extable_len) 305 325 fp->aux->extable = (void *)fimage + FUNCTION_DESCR_SIZE + proglen + fixup_len; ··· 316 340 extra_pass)) { 317 341 bpf_arch_text_copy(&fhdr->size, &hdr->size, sizeof(hdr->size)); 318 342 bpf_jit_binary_pack_free(fhdr, hdr); 319 - fp = org_fp; 320 - goto out_addrs; 343 + goto out_err; 321 344 } 322 345 bpf_jit_build_epilogue(code_base, &cgctx); 323 346 ··· 338 363 ((u64 *)image)[1] = local_paca->kernel_toc; 339 364 #endif 340 365 366 + if (!fp->is_func || extra_pass) { 367 + if (bpf_jit_binary_pack_finalize(fhdr, hdr)) 368 + goto out_err; 369 + } 370 + 341 371 fp->bpf_func = (void *)fimage; 342 372 fp->jited = 1; 343 373 fp->jited_len = cgctx.idx * 4 + FUNCTION_DESCR_SIZE; 344 374 345 375 if (!fp->is_func || extra_pass) { 346 - if (bpf_jit_binary_pack_finalize(fhdr, hdr)) { 347 - fp = org_fp; 348 - goto out_addrs; 349 - } 350 376 bpf_prog_fill_jited_linfo(fp, addrs); 351 377 /* 352 378 * On ABI V1, executable code starts after the function ··· 374 398 jit_data->hdr = hdr; 375 399 } 376 400 377 - out: 378 - if (bpf_blinded) 379 - bpf_jit_prog_release_other(fp, fp == org_fp ? tmp_fp : org_fp); 380 - 381 401 return fp; 402 + 403 + out_err: 404 + if (extra_pass) { 405 + fp->bpf_func = NULL; 406 + fp->jited = 0; 407 + fp->jited_len = 0; 408 + } 409 + goto out_addrs; 382 410 } 383 411 384 412 /*
+23 -40
arch/riscv/net/bpf_jit_core.c
··· 41 41 return true; 42 42 } 43 43 44 - struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 44 + struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog) 45 45 { 46 46 unsigned int prog_size = 0, extable_size = 0; 47 - bool tmp_blinded = false, extra_pass = false; 48 - struct bpf_prog *tmp, *orig_prog = prog; 47 + bool extra_pass = false; 49 48 int pass = 0, prev_ninsns = 0, i; 50 49 struct rv_jit_data *jit_data; 51 50 struct rv_jit_context *ctx; 52 51 53 52 if (!prog->jit_requested) 54 - return orig_prog; 55 - 56 - tmp = bpf_jit_blind_constants(prog); 57 - if (IS_ERR(tmp)) 58 - return orig_prog; 59 - if (tmp != prog) { 60 - tmp_blinded = true; 61 - prog = tmp; 62 - } 53 + return prog; 63 54 64 55 jit_data = prog->aux->jit_data; 65 56 if (!jit_data) { 66 57 jit_data = kzalloc_obj(*jit_data); 67 58 if (!jit_data) { 68 - prog = orig_prog; 69 - goto out; 59 + return prog; 70 60 } 71 61 prog->aux->jit_data = jit_data; 72 62 } ··· 73 83 ctx->user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena); 74 84 ctx->prog = prog; 75 85 ctx->offset = kzalloc_objs(int, prog->len); 76 - if (!ctx->offset) { 77 - prog = orig_prog; 86 + if (!ctx->offset) 78 87 goto out_offset; 79 - } 80 88 81 - if (build_body(ctx, extra_pass, NULL)) { 82 - prog = orig_prog; 89 + if (build_body(ctx, extra_pass, NULL)) 83 90 goto out_offset; 84 - } 85 91 86 92 for (i = 0; i < prog->len; i++) { 87 93 prev_ninsns += 32; ··· 91 105 bpf_jit_build_prologue(ctx, bpf_is_subprog(prog)); 92 106 ctx->prologue_len = ctx->ninsns; 93 107 94 - if (build_body(ctx, extra_pass, ctx->offset)) { 95 - prog = orig_prog; 108 + if (build_body(ctx, extra_pass, ctx->offset)) 96 109 goto out_offset; 97 - } 98 110 99 111 ctx->epilogue_offset = ctx->ninsns; 100 112 bpf_jit_build_epilogue(ctx); ··· 110 126 &jit_data->ro_image, sizeof(u32), 111 127 &jit_data->header, &jit_data->image, 112 128 bpf_fill_ill_insns); 113 - if (!jit_data->ro_header) { 114 - prog = orig_prog; 129 + if (!jit_data->ro_header) 115 130 goto out_offset; 116 - } 117 131 118 132 /* 119 133 * Use the image(RW) for writing the JITed instructions. But also save ··· 132 150 133 151 if (i == NR_JIT_ITERATIONS) { 134 152 pr_err("bpf-jit: image did not converge in <%d passes!\n", i); 135 - prog = orig_prog; 136 153 goto out_free_hdr; 137 154 } 138 155 ··· 144 163 ctx->nexentries = 0; 145 164 146 165 bpf_jit_build_prologue(ctx, bpf_is_subprog(prog)); 147 - if (build_body(ctx, extra_pass, NULL)) { 148 - prog = orig_prog; 166 + if (build_body(ctx, extra_pass, NULL)) 149 167 goto out_free_hdr; 150 - } 151 168 bpf_jit_build_epilogue(ctx); 152 169 153 170 if (bpf_jit_enable > 1) 154 171 bpf_jit_dump(prog->len, prog_size, pass, ctx->insns); 172 + 173 + if (!prog->is_func || extra_pass) { 174 + if (WARN_ON(bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->header))) { 175 + /* ro_header has been freed */ 176 + jit_data->ro_header = NULL; 177 + jit_data->header = NULL; 178 + goto out_free_hdr; 179 + } 180 + } 155 181 156 182 prog->bpf_func = (void *)ctx->ro_insns + cfi_get_offset(); 157 183 prog->jited = 1; 158 184 prog->jited_len = prog_size - cfi_get_offset(); 159 185 160 186 if (!prog->is_func || extra_pass) { 161 - if (WARN_ON(bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->header))) { 162 - /* ro_header has been freed */ 163 - jit_data->ro_header = NULL; 164 - prog = orig_prog; 165 - goto out_offset; 166 - } 167 187 for (i = 0; i < prog->len; i++) 168 188 ctx->offset[i] = ninsns_rvoff(ctx->offset[i]); 169 189 bpf_prog_fill_jited_linfo(prog, ctx->offset); ··· 173 191 kfree(jit_data); 174 192 prog->aux->jit_data = NULL; 175 193 } 176 - out: 177 194 178 - if (tmp_blinded) 179 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 180 - tmp : orig_prog); 181 195 return prog; 182 196 183 197 out_free_hdr: 198 + if (extra_pass) { 199 + prog->bpf_func = NULL; 200 + prog->jited = 0; 201 + prog->jited_len = 0; 202 + } 184 203 if (jit_data->header) { 185 204 bpf_arch_text_copy(&jit_data->ro_header->size, &jit_data->header->size, 186 205 sizeof(jit_data->header->size));
+21 -40
arch/s390/net/bpf_jit_comp.c
··· 2312 2312 /* 2313 2313 * Compile eBPF program "fp" 2314 2314 */ 2315 - struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) 2315 + struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *fp) 2316 2316 { 2317 - struct bpf_prog *tmp, *orig_fp = fp; 2318 2317 struct bpf_binary_header *header; 2319 2318 struct s390_jit_data *jit_data; 2320 - bool tmp_blinded = false; 2321 2319 bool extra_pass = false; 2322 2320 struct bpf_jit jit; 2323 2321 int pass; 2324 2322 2325 2323 if (!fp->jit_requested) 2326 - return orig_fp; 2327 - 2328 - tmp = bpf_jit_blind_constants(fp); 2329 - /* 2330 - * If blinding was requested and we failed during blinding, 2331 - * we must fall back to the interpreter. 2332 - */ 2333 - if (IS_ERR(tmp)) 2334 - return orig_fp; 2335 - if (tmp != fp) { 2336 - tmp_blinded = true; 2337 - fp = tmp; 2338 - } 2324 + return fp; 2339 2325 2340 2326 jit_data = fp->aux->jit_data; 2341 2327 if (!jit_data) { 2342 2328 jit_data = kzalloc_obj(*jit_data); 2343 - if (!jit_data) { 2344 - fp = orig_fp; 2345 - goto out; 2346 - } 2329 + if (!jit_data) 2330 + return fp; 2347 2331 fp->aux->jit_data = jit_data; 2348 2332 } 2349 2333 if (jit_data->ctx.addrs) { ··· 2340 2356 2341 2357 memset(&jit, 0, sizeof(jit)); 2342 2358 jit.addrs = kvcalloc(fp->len + 1, sizeof(*jit.addrs), GFP_KERNEL); 2343 - if (jit.addrs == NULL) { 2344 - fp = orig_fp; 2345 - goto free_addrs; 2346 - } 2359 + if (jit.addrs == NULL) 2360 + goto out_err; 2347 2361 /* 2348 2362 * Three initial passes: 2349 2363 * - 1/2: Determine clobbered registers 2350 2364 * - 3: Calculate program size and addrs array 2351 2365 */ 2352 2366 for (pass = 1; pass <= 3; pass++) { 2353 - if (bpf_jit_prog(&jit, fp, extra_pass)) { 2354 - fp = orig_fp; 2355 - goto free_addrs; 2356 - } 2367 + if (bpf_jit_prog(&jit, fp, extra_pass)) 2368 + goto out_err; 2357 2369 } 2358 2370 /* 2359 2371 * Final pass: Allocate and generate program 2360 2372 */ 2361 2373 header = bpf_jit_alloc(&jit, fp); 2362 - if (!header) { 2363 - fp = orig_fp; 2364 - goto free_addrs; 2365 - } 2374 + if (!header) 2375 + goto out_err; 2366 2376 skip_init_ctx: 2367 2377 if (bpf_jit_prog(&jit, fp, extra_pass)) { 2368 2378 bpf_jit_binary_free(header); 2369 - fp = orig_fp; 2370 - goto free_addrs; 2379 + goto out_err; 2371 2380 } 2372 2381 if (bpf_jit_enable > 1) { 2373 2382 bpf_jit_dump(fp->len, jit.size, pass, jit.prg_buf); ··· 2369 2392 if (!fp->is_func || extra_pass) { 2370 2393 if (bpf_jit_binary_lock_ro(header)) { 2371 2394 bpf_jit_binary_free(header); 2372 - fp = orig_fp; 2373 - goto free_addrs; 2395 + goto out_err; 2374 2396 } 2375 2397 } else { 2376 2398 jit_data->header = header; ··· 2387 2411 kfree(jit_data); 2388 2412 fp->aux->jit_data = NULL; 2389 2413 } 2390 - out: 2391 - if (tmp_blinded) 2392 - bpf_jit_prog_release_other(fp, fp == orig_fp ? 2393 - tmp : orig_fp); 2414 + 2394 2415 return fp; 2416 + 2417 + out_err: 2418 + if (extra_pass) { 2419 + fp->bpf_func = NULL; 2420 + fp->jited = 0; 2421 + fp->jited_len = 0; 2422 + } 2423 + goto free_addrs; 2395 2424 } 2396 2425 2397 2426 bool bpf_jit_supports_kfunc_call(void)
+22 -41
arch/sparc/net/bpf_jit_comp_64.c
··· 1477 1477 struct jit_ctx ctx; 1478 1478 }; 1479 1479 1480 - struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 1480 + struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog) 1481 1481 { 1482 - struct bpf_prog *tmp, *orig_prog = prog; 1483 1482 struct sparc64_jit_data *jit_data; 1484 1483 struct bpf_binary_header *header; 1485 1484 u32 prev_image_size, image_size; 1486 - bool tmp_blinded = false; 1487 1485 bool extra_pass = false; 1488 1486 struct jit_ctx ctx; 1489 1487 u8 *image_ptr; 1490 1488 int pass, i; 1491 1489 1492 1490 if (!prog->jit_requested) 1493 - return orig_prog; 1494 - 1495 - tmp = bpf_jit_blind_constants(prog); 1496 - /* If blinding was requested and we failed during blinding, 1497 - * we must fall back to the interpreter. 1498 - */ 1499 - if (IS_ERR(tmp)) 1500 - return orig_prog; 1501 - if (tmp != prog) { 1502 - tmp_blinded = true; 1503 - prog = tmp; 1504 - } 1491 + return prog; 1505 1492 1506 1493 jit_data = prog->aux->jit_data; 1507 1494 if (!jit_data) { 1508 1495 jit_data = kzalloc_obj(*jit_data); 1509 - if (!jit_data) { 1510 - prog = orig_prog; 1511 - goto out; 1512 - } 1496 + if (!jit_data) 1497 + return prog; 1513 1498 prog->aux->jit_data = jit_data; 1514 1499 } 1515 1500 if (jit_data->ctx.offset) { ··· 1512 1527 ctx.prog = prog; 1513 1528 1514 1529 ctx.offset = kmalloc_array(prog->len, sizeof(unsigned int), GFP_KERNEL); 1515 - if (ctx.offset == NULL) { 1516 - prog = orig_prog; 1517 - goto out_off; 1518 - } 1530 + if (ctx.offset == NULL) 1531 + goto out_err; 1519 1532 1520 1533 /* Longest sequence emitted is for bswap32, 12 instructions. Pre-cook 1521 1534 * the offset array so that we converge faster. ··· 1526 1543 ctx.idx = 0; 1527 1544 1528 1545 build_prologue(&ctx); 1529 - if (build_body(&ctx)) { 1530 - prog = orig_prog; 1531 - goto out_off; 1532 - } 1546 + if (build_body(&ctx)) 1547 + goto out_err; 1533 1548 build_epilogue(&ctx); 1534 1549 1535 1550 if (bpf_jit_enable > 1) ··· 1550 1569 image_size = sizeof(u32) * ctx.idx; 1551 1570 header = bpf_jit_binary_alloc(image_size, &image_ptr, 1552 1571 sizeof(u32), jit_fill_hole); 1553 - if (header == NULL) { 1554 - prog = orig_prog; 1555 - goto out_off; 1556 - } 1572 + if (header == NULL) 1573 + goto out_err; 1557 1574 1558 1575 ctx.image = (u32 *)image_ptr; 1559 1576 skip_init_ctx: ··· 1561 1582 1562 1583 if (build_body(&ctx)) { 1563 1584 bpf_jit_binary_free(header); 1564 - prog = orig_prog; 1565 - goto out_off; 1585 + goto out_err; 1566 1586 } 1567 1587 1568 1588 build_epilogue(&ctx); ··· 1570 1592 pr_err("bpf_jit: Failed to converge, prev_size=%u size=%d\n", 1571 1593 prev_image_size, ctx.idx * 4); 1572 1594 bpf_jit_binary_free(header); 1573 - prog = orig_prog; 1574 - goto out_off; 1595 + goto out_err; 1575 1596 } 1576 1597 1577 1598 if (bpf_jit_enable > 1) ··· 1581 1604 if (!prog->is_func || extra_pass) { 1582 1605 if (bpf_jit_binary_lock_ro(header)) { 1583 1606 bpf_jit_binary_free(header); 1584 - prog = orig_prog; 1585 - goto out_off; 1607 + goto out_err; 1586 1608 } 1587 1609 } else { 1588 1610 jit_data->ctx = ctx; ··· 1600 1624 kfree(jit_data); 1601 1625 prog->aux->jit_data = NULL; 1602 1626 } 1603 - out: 1604 - if (tmp_blinded) 1605 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 1606 - tmp : orig_prog); 1627 + 1607 1628 return prog; 1629 + 1630 + out_err: 1631 + if (extra_pass) { 1632 + prog->bpf_func = NULL; 1633 + prog->jited = 0; 1634 + prog->jited_len = 0; 1635 + } 1636 + goto out_off; 1608 1637 }
+23 -50
arch/x86/net/bpf_jit_comp.c
··· 58 58 #define EMIT_ENDBR() EMIT(gen_endbr(), 4) 59 59 #define EMIT_ENDBR_POISON() EMIT(gen_endbr_poison(), 4) 60 60 #else 61 - #define EMIT_ENDBR() 62 - #define EMIT_ENDBR_POISON() 61 + #define EMIT_ENDBR() do { } while (0) 62 + #define EMIT_ENDBR_POISON() do { } while (0) 63 63 #endif 64 64 65 65 static bool is_imm8(int value) ··· 1649 1649 return 0; 1650 1650 } 1651 1651 1652 - static int do_jit(struct bpf_prog *bpf_prog, int *addrs, u8 *image, u8 *rw_image, 1653 - int oldproglen, struct jit_context *ctx, bool jmp_padding) 1652 + static int do_jit(struct bpf_verifier_env *env, struct bpf_prog *bpf_prog, int *addrs, u8 *image, 1653 + u8 *rw_image, int oldproglen, struct jit_context *ctx, bool jmp_padding) 1654 1654 { 1655 1655 bool tail_call_reachable = bpf_prog->aux->tail_call_reachable; 1656 1656 struct bpf_insn *insn = bpf_prog->insnsi; ··· 1663 1663 void __percpu *priv_stack_ptr; 1664 1664 int i, excnt = 0; 1665 1665 int ilen, proglen = 0; 1666 - u8 *prog = temp; 1666 + u8 *ip, *prog = temp; 1667 1667 u32 stack_depth; 1668 1668 int err; 1669 1669 ··· 1733 1733 if (dst_reg == BPF_REG_FP) 1734 1734 dst_reg = X86_REG_R9; 1735 1735 } 1736 + 1737 + if (bpf_insn_is_indirect_target(env, bpf_prog, i - 1)) 1738 + EMIT_ENDBR(); 1739 + 1740 + ip = image + addrs[i - 1] + (prog - temp); 1736 1741 1737 1742 switch (insn->code) { 1738 1743 /* ALU */ ··· 2445 2440 2446 2441 /* call */ 2447 2442 case BPF_JMP | BPF_CALL: { 2448 - u8 *ip = image + addrs[i - 1]; 2449 - 2450 2443 func = (u8 *) __bpf_call_base + imm32; 2451 2444 if (src_reg == BPF_PSEUDO_CALL && tail_call_reachable) { 2452 2445 LOAD_TAIL_CALL_CNT_PTR(stack_depth); ··· 2468 2465 if (imm32) 2469 2466 emit_bpf_tail_call_direct(bpf_prog, 2470 2467 &bpf_prog->aux->poke_tab[imm32 - 1], 2471 - &prog, image + addrs[i - 1], 2468 + &prog, 2469 + ip, 2472 2470 callee_regs_used, 2473 2471 stack_depth, 2474 2472 ctx); ··· 2478 2474 &prog, 2479 2475 callee_regs_used, 2480 2476 stack_depth, 2481 - image + addrs[i - 1], 2477 + ip, 2482 2478 ctx); 2483 2479 break; 2484 2480 ··· 2643 2639 break; 2644 2640 2645 2641 case BPF_JMP | BPF_JA | BPF_X: 2646 - emit_indirect_jump(&prog, insn->dst_reg, image + addrs[i - 1]); 2642 + emit_indirect_jump(&prog, insn->dst_reg, ip); 2647 2643 break; 2648 2644 case BPF_JMP | BPF_JA: 2649 2645 case BPF_JMP32 | BPF_JA: ··· 2733 2729 ctx->cleanup_addr = proglen; 2734 2730 if (bpf_prog_was_classic(bpf_prog) && 2735 2731 !ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN)) { 2736 - u8 *ip = image + addrs[i - 1]; 2737 - 2738 2732 if (emit_spectre_bhb_barrier(&prog, ip, bpf_prog)) 2739 2733 return -EINVAL; 2740 2734 } ··· 3715 3713 #define MAX_PASSES 20 3716 3714 #define PADDING_PASSES (MAX_PASSES - 5) 3717 3715 3718 - struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 3716 + struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog) 3719 3717 { 3720 3718 struct bpf_binary_header *rw_header = NULL; 3721 3719 struct bpf_binary_header *header = NULL; 3722 - struct bpf_prog *tmp, *orig_prog = prog; 3723 3720 void __percpu *priv_stack_ptr = NULL; 3724 3721 struct x64_jit_data *jit_data; 3725 3722 int priv_stack_alloc_sz; 3726 3723 int proglen, oldproglen = 0; 3727 3724 struct jit_context ctx = {}; 3728 - bool tmp_blinded = false; 3729 3725 bool extra_pass = false; 3730 3726 bool padding = false; 3731 3727 u8 *rw_image = NULL; ··· 3733 3733 int i; 3734 3734 3735 3735 if (!prog->jit_requested) 3736 - return orig_prog; 3737 - 3738 - tmp = bpf_jit_blind_constants(prog); 3739 - /* 3740 - * If blinding was requested and we failed during blinding, 3741 - * we must fall back to the interpreter. 3742 - */ 3743 - if (IS_ERR(tmp)) 3744 - return orig_prog; 3745 - if (tmp != prog) { 3746 - tmp_blinded = true; 3747 - prog = tmp; 3748 - } 3736 + return prog; 3749 3737 3750 3738 jit_data = prog->aux->jit_data; 3751 3739 if (!jit_data) { 3752 3740 jit_data = kzalloc_obj(*jit_data); 3753 - if (!jit_data) { 3754 - prog = orig_prog; 3755 - goto out; 3756 - } 3741 + if (!jit_data) 3742 + return prog; 3757 3743 prog->aux->jit_data = jit_data; 3758 3744 } 3759 3745 priv_stack_ptr = prog->aux->priv_stack_ptr; ··· 3751 3765 priv_stack_alloc_sz = round_up(prog->aux->stack_depth, 8) + 3752 3766 2 * PRIV_STACK_GUARD_SZ; 3753 3767 priv_stack_ptr = __alloc_percpu_gfp(priv_stack_alloc_sz, 8, GFP_KERNEL); 3754 - if (!priv_stack_ptr) { 3755 - prog = orig_prog; 3768 + if (!priv_stack_ptr) 3756 3769 goto out_priv_stack; 3757 - } 3758 3770 3759 3771 priv_stack_init_guard(priv_stack_ptr, priv_stack_alloc_sz); 3760 3772 prog->aux->priv_stack_ptr = priv_stack_ptr; ··· 3770 3786 goto skip_init_addrs; 3771 3787 } 3772 3788 addrs = kvmalloc_objs(*addrs, prog->len + 1); 3773 - if (!addrs) { 3774 - prog = orig_prog; 3789 + if (!addrs) 3775 3790 goto out_addrs; 3776 - } 3777 3791 3778 3792 /* 3779 3793 * Before first pass, make a rough estimation of addrs[] ··· 3793 3811 for (pass = 0; pass < MAX_PASSES || image; pass++) { 3794 3812 if (!padding && pass >= PADDING_PASSES) 3795 3813 padding = true; 3796 - proglen = do_jit(prog, addrs, image, rw_image, oldproglen, &ctx, padding); 3814 + proglen = do_jit(env, prog, addrs, image, rw_image, oldproglen, &ctx, padding); 3797 3815 if (proglen <= 0) { 3798 3816 out_image: 3799 3817 image = NULL; ··· 3802 3820 sizeof(rw_header->size)); 3803 3821 bpf_jit_binary_pack_free(header, rw_header); 3804 3822 } 3805 - /* Fall back to interpreter mode */ 3806 - prog = orig_prog; 3807 3823 if (extra_pass) { 3808 3824 prog->bpf_func = NULL; 3809 3825 prog->jited = 0; ··· 3832 3852 header = bpf_jit_binary_pack_alloc(roundup(proglen, align) + extable_size, 3833 3853 &image, align, &rw_header, &rw_image, 3834 3854 jit_fill_hole); 3835 - if (!header) { 3836 - prog = orig_prog; 3855 + if (!header) 3837 3856 goto out_addrs; 3838 - } 3839 3857 prog->aux->extable = (void *) image + roundup(proglen, align); 3840 3858 } 3841 3859 oldproglen = proglen; ··· 3886 3908 prog->bpf_func = (void *)image + cfi_get_offset(); 3887 3909 prog->jited = 1; 3888 3910 prog->jited_len = proglen - cfi_get_offset(); 3889 - } else { 3890 - prog = orig_prog; 3891 3911 } 3892 3912 3893 3913 if (!image || !prog->is_func || extra_pass) { ··· 3901 3925 kfree(jit_data); 3902 3926 prog->aux->jit_data = NULL; 3903 3927 } 3904 - out: 3905 - if (tmp_blinded) 3906 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 3907 - tmp : orig_prog); 3928 + 3908 3929 return prog; 3909 3930 } 3910 3931
+5 -30
arch/x86/net/bpf_jit_comp32.c
··· 2518 2518 return true; 2519 2519 } 2520 2520 2521 - struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 2521 + struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog) 2522 2522 { 2523 2523 struct bpf_binary_header *header = NULL; 2524 - struct bpf_prog *tmp, *orig_prog = prog; 2525 2524 int proglen, oldproglen = 0; 2526 2525 struct jit_context ctx = {}; 2527 - bool tmp_blinded = false; 2528 2526 u8 *image = NULL; 2529 2527 int *addrs; 2530 2528 int pass; 2531 2529 int i; 2532 2530 2533 2531 if (!prog->jit_requested) 2534 - return orig_prog; 2535 - 2536 - tmp = bpf_jit_blind_constants(prog); 2537 - /* 2538 - * If blinding was requested and we failed during blinding, 2539 - * we must fall back to the interpreter. 2540 - */ 2541 - if (IS_ERR(tmp)) 2542 - return orig_prog; 2543 - if (tmp != prog) { 2544 - tmp_blinded = true; 2545 - prog = tmp; 2546 - } 2532 + return prog; 2547 2533 2548 2534 addrs = kmalloc_objs(*addrs, prog->len); 2549 - if (!addrs) { 2550 - prog = orig_prog; 2551 - goto out; 2552 - } 2535 + if (!addrs) 2536 + return prog; 2553 2537 2554 2538 /* 2555 2539 * Before first pass, make a rough estimation of addrs[] ··· 2558 2574 image = NULL; 2559 2575 if (header) 2560 2576 bpf_jit_binary_free(header); 2561 - prog = orig_prog; 2562 2577 goto out_addrs; 2563 2578 } 2564 2579 if (image) { ··· 2571 2588 if (proglen == oldproglen) { 2572 2589 header = bpf_jit_binary_alloc(proglen, &image, 2573 2590 1, jit_fill_hole); 2574 - if (!header) { 2575 - prog = orig_prog; 2591 + if (!header) 2576 2592 goto out_addrs; 2577 - } 2578 2593 } 2579 2594 oldproglen = proglen; 2580 2595 cond_resched(); ··· 2585 2604 prog->bpf_func = (void *)image; 2586 2605 prog->jited = 1; 2587 2606 prog->jited_len = proglen; 2588 - } else { 2589 - prog = orig_prog; 2590 2607 } 2591 2608 2592 2609 out_addrs: 2593 2610 kfree(addrs); 2594 - out: 2595 - if (tmp_blinded) 2596 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 2597 - tmp : orig_prog); 2598 2611 return prog; 2599 2612 } 2600 2613
+2
include/linux/bpf.h
··· 1541 1541 int bpf_jit_charge_modmem(u32 size); 1542 1542 void bpf_jit_uncharge_modmem(u32 size); 1543 1543 bool bpf_prog_has_trampoline(const struct bpf_prog *prog); 1544 + bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog, 1545 + int insn_idx); 1544 1546 #else 1545 1547 static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link, 1546 1548 struct bpf_trampoline *tr,
+5 -4
include/linux/bpf_verifier.h
··· 630 630 631 631 /* below fields are initialized once */ 632 632 unsigned int orig_idx; /* original instruction index */ 633 - bool jmp_point; 634 - bool prune_point; 633 + u32 jmp_point:1; 634 + u32 prune_point:1; 635 635 /* ensure we check state equivalence and save state checkpoint and 636 636 * this instruction, regardless of any heuristics 637 637 */ 638 - bool force_checkpoint; 638 + u32 force_checkpoint:1; 639 639 /* true if instruction is a call to a helper function that 640 640 * accepts callback function as a parameter. 641 641 */ 642 - bool calls_callback; 642 + u32 calls_callback:1; 643 + u32 indirect_target:1; /* if it is an indirect jump target */ 643 644 /* 644 645 * CFG strongly connected component this instruction belongs to, 645 646 * zero if it is a singleton SCC.
+48 -2
include/linux/filter.h
··· 1108 1108 return sk_filter_trim_cap(sk, skb, 1); 1109 1109 } 1110 1110 1111 + struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct bpf_prog *fp, 1112 + int *err); 1111 1113 struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err); 1112 1114 void bpf_prog_free(struct bpf_prog *fp); 1113 1115 ··· 1155 1153 ((u64 (*)(u64, u64, u64, u64, u64, const struct bpf_insn *)) \ 1156 1154 (void *)__bpf_call_base) 1157 1155 1158 - struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog); 1156 + struct bpf_prog *bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog); 1159 1157 void bpf_jit_compile(struct bpf_prog *prog); 1160 1158 bool bpf_jit_needs_zext(void); 1161 1159 bool bpf_jit_inlines_helper_call(s32 imm); ··· 1186 1184 1187 1185 struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off, 1188 1186 const struct bpf_insn *patch, u32 len); 1187 + 1188 + #ifdef CONFIG_BPF_SYSCALL 1189 + struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off, 1190 + const struct bpf_insn *patch, u32 len); 1191 + struct bpf_insn_aux_data *bpf_dup_insn_aux_data(struct bpf_verifier_env *env); 1192 + void bpf_restore_insn_aux_data(struct bpf_verifier_env *env, 1193 + struct bpf_insn_aux_data *orig_insn_aux); 1194 + #else 1195 + static inline struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off, 1196 + const struct bpf_insn *patch, u32 len) 1197 + { 1198 + return ERR_PTR(-ENOTSUPP); 1199 + } 1200 + 1201 + static inline struct bpf_insn_aux_data *bpf_dup_insn_aux_data(struct bpf_verifier_env *env) 1202 + { 1203 + return NULL; 1204 + } 1205 + 1206 + static inline void bpf_restore_insn_aux_data(struct bpf_verifier_env *env, 1207 + struct bpf_insn_aux_data *orig_insn_aux) 1208 + { 1209 + } 1210 + #endif /* CONFIG_BPF_SYSCALL */ 1211 + 1189 1212 int bpf_remove_insns(struct bpf_prog *prog, u32 off, u32 cnt); 1190 1213 1191 1214 static inline bool xdp_return_frame_no_direct(void) ··· 1337 1310 1338 1311 const char *bpf_jit_get_prog_name(struct bpf_prog *prog); 1339 1312 1340 - struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *fp); 1313 + struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, struct bpf_prog *prog); 1341 1314 void bpf_jit_prog_release_other(struct bpf_prog *fp, struct bpf_prog *fp_other); 1315 + 1316 + static inline bool bpf_prog_need_blind(const struct bpf_prog *prog) 1317 + { 1318 + return prog->blinding_requested && !prog->blinded; 1319 + } 1342 1320 1343 1321 static inline void bpf_jit_dump(unsigned int flen, unsigned int proglen, 1344 1322 u32 pass, void *image) ··· 1483 1451 { 1484 1452 } 1485 1453 1454 + static inline bool bpf_prog_need_blind(const struct bpf_prog *prog) 1455 + { 1456 + return false; 1457 + } 1458 + 1459 + static inline 1460 + struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, struct bpf_prog *prog) 1461 + { 1462 + return prog; 1463 + } 1464 + 1465 + static inline void bpf_jit_prog_release_other(struct bpf_prog *fp, struct bpf_prog *fp_other) 1466 + { 1467 + } 1486 1468 #endif /* CONFIG_BPF_JIT */ 1487 1469 1488 1470 void bpf_prog_kallsyms_del_all(struct bpf_prog *fp);
+99 -39
kernel/bpf/core.c
··· 1491 1491 bpf_prog_clone_free(fp_other); 1492 1492 } 1493 1493 1494 - static void adjust_insn_arrays(struct bpf_prog *prog, u32 off, u32 len) 1495 - { 1496 - #ifdef CONFIG_BPF_SYSCALL 1497 - struct bpf_map *map; 1498 - int i; 1499 - 1500 - if (len <= 1) 1501 - return; 1502 - 1503 - for (i = 0; i < prog->aux->used_map_cnt; i++) { 1504 - map = prog->aux->used_maps[i]; 1505 - if (map->map_type == BPF_MAP_TYPE_INSN_ARRAY) 1506 - bpf_insn_array_adjust(map, off, len); 1507 - } 1508 - #endif 1509 - } 1510 - 1511 - struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog) 1494 + /* 1495 + * Now this function is used only to blind the main prog and must be invoked only when 1496 + * bpf_prog_need_blind() returns true. 1497 + */ 1498 + struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, struct bpf_prog *prog) 1512 1499 { 1513 1500 struct bpf_insn insn_buff[16], aux[2]; 1514 1501 struct bpf_prog *clone, *tmp; ··· 1503 1516 struct bpf_insn *insn; 1504 1517 int i, rewritten; 1505 1518 1506 - if (!prog->blinding_requested || prog->blinded) 1507 - return prog; 1519 + if (WARN_ON_ONCE(env && env->prog != prog)) 1520 + return ERR_PTR(-EINVAL); 1508 1521 1509 1522 clone = bpf_prog_clone_create(prog, GFP_USER); 1510 1523 if (!clone) 1511 1524 return ERR_PTR(-ENOMEM); 1525 + 1526 + /* make sure bpf_patch_insn_data() patches the correct prog */ 1527 + if (env) 1528 + env->prog = clone; 1512 1529 1513 1530 insn_cnt = clone->len; 1514 1531 insn = clone->insnsi; ··· 1541 1550 if (!rewritten) 1542 1551 continue; 1543 1552 1544 - tmp = bpf_patch_insn_single(clone, i, insn_buff, rewritten); 1545 - if (IS_ERR(tmp)) { 1553 + if (env) 1554 + tmp = bpf_patch_insn_data(env, i, insn_buff, rewritten); 1555 + else 1556 + tmp = bpf_patch_insn_single(clone, i, insn_buff, rewritten); 1557 + 1558 + if (IS_ERR_OR_NULL(tmp)) { 1559 + if (env) 1560 + /* restore the original prog */ 1561 + env->prog = prog; 1546 1562 /* Patching may have repointed aux->prog during 1547 1563 * realloc from the original one, so we need to 1548 1564 * fix it up here on error. 1549 1565 */ 1550 1566 bpf_jit_prog_release_other(prog, clone); 1551 - return tmp; 1567 + return IS_ERR(tmp) ? tmp : ERR_PTR(-ENOMEM); 1552 1568 } 1553 1569 1554 1570 clone = tmp; 1555 1571 insn_delta = rewritten - 1; 1556 1572 1557 - /* Instructions arrays must be updated using absolute xlated offsets */ 1558 - adjust_insn_arrays(clone, prog->aux->subprog_start + i, rewritten); 1573 + if (env) 1574 + env->prog = clone; 1559 1575 1560 1576 /* Walk new program and skip insns we just inserted. */ 1561 1577 insn = clone->insnsi + i + insn_delta; ··· 1572 1574 1573 1575 clone->blinded = 1; 1574 1576 return clone; 1577 + } 1578 + 1579 + bool bpf_insn_is_indirect_target(const struct bpf_verifier_env *env, const struct bpf_prog *prog, 1580 + int insn_idx) 1581 + { 1582 + if (!env) 1583 + return false; 1584 + insn_idx += prog->aux->subprog_start; 1585 + return env->insn_aux_data[insn_idx].indirect_target; 1575 1586 } 1576 1587 #endif /* CONFIG_BPF_JIT */ 1577 1588 ··· 2540 2533 return select_interpreter; 2541 2534 } 2542 2535 2543 - /** 2544 - * bpf_prog_select_runtime - select exec runtime for BPF program 2545 - * @fp: bpf_prog populated with BPF program 2546 - * @err: pointer to error variable 2547 - * 2548 - * Try to JIT eBPF program, if JIT is not available, use interpreter. 2549 - * The BPF program will be executed via bpf_prog_run() function. 2550 - * 2551 - * Return: the &fp argument along with &err set to 0 for success or 2552 - * a negative errno code on failure 2553 - */ 2554 - struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err) 2536 + static struct bpf_prog *bpf_prog_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog) 2537 + { 2538 + #ifdef CONFIG_BPF_JIT 2539 + struct bpf_prog *orig_prog; 2540 + struct bpf_insn_aux_data *orig_insn_aux; 2541 + 2542 + if (!bpf_prog_need_blind(prog)) 2543 + return bpf_int_jit_compile(env, prog); 2544 + 2545 + if (env) { 2546 + /* 2547 + * If env is not NULL, we are called from the end of bpf_check(), at this 2548 + * point, only insn_aux_data is used after failure, so it should be restored 2549 + * on failure. 2550 + */ 2551 + orig_insn_aux = bpf_dup_insn_aux_data(env); 2552 + if (!orig_insn_aux) 2553 + return prog; 2554 + } 2555 + 2556 + orig_prog = prog; 2557 + prog = bpf_jit_blind_constants(env, prog); 2558 + /* 2559 + * If blinding was requested and we failed during blinding, we must fall 2560 + * back to the interpreter. 2561 + */ 2562 + if (IS_ERR(prog)) 2563 + goto out_restore; 2564 + 2565 + prog = bpf_int_jit_compile(env, prog); 2566 + if (prog->jited) { 2567 + bpf_jit_prog_release_other(prog, orig_prog); 2568 + if (env) 2569 + vfree(orig_insn_aux); 2570 + return prog; 2571 + } 2572 + 2573 + bpf_jit_prog_release_other(orig_prog, prog); 2574 + 2575 + out_restore: 2576 + prog = orig_prog; 2577 + if (env) 2578 + bpf_restore_insn_aux_data(env, orig_insn_aux); 2579 + #endif 2580 + return prog; 2581 + } 2582 + 2583 + struct bpf_prog *__bpf_prog_select_runtime(struct bpf_verifier_env *env, struct bpf_prog *fp, 2584 + int *err) 2555 2585 { 2556 2586 /* In case of BPF to BPF calls, verifier did all the prep 2557 2587 * work with regards to JITing, etc. ··· 2616 2572 if (*err) 2617 2573 return fp; 2618 2574 2619 - fp = bpf_int_jit_compile(fp); 2575 + fp = bpf_prog_jit_compile(env, fp); 2620 2576 bpf_prog_jit_attempt_done(fp); 2621 2577 if (!fp->jited && jit_needed) { 2622 2578 *err = -ENOTSUPP; ··· 2641 2597 *err = bpf_check_tail_call(fp); 2642 2598 2643 2599 return fp; 2600 + } 2601 + 2602 + /** 2603 + * bpf_prog_select_runtime - select exec runtime for BPF program 2604 + * @fp: bpf_prog populated with BPF program 2605 + * @err: pointer to error variable 2606 + * 2607 + * Try to JIT eBPF program, if JIT is not available, use interpreter. 2608 + * The BPF program will be executed via bpf_prog_run() function. 2609 + * 2610 + * Return: the &fp argument along with &err set to 0 for success or 2611 + * a negative errno code on failure 2612 + */ 2613 + struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err) 2614 + { 2615 + return __bpf_prog_select_runtime(NULL, fp, err); 2644 2616 } 2645 2617 EXPORT_SYMBOL_GPL(bpf_prog_select_runtime); 2646 2618 ··· 3145 3085 * It is encouraged to implement bpf_int_jit_compile() instead, so that 3146 3086 * eBPF and implicitly also cBPF can get JITed! 3147 3087 */ 3148 - struct bpf_prog * __weak bpf_int_jit_compile(struct bpf_prog *prog) 3088 + struct bpf_prog * __weak bpf_int_jit_compile(struct bpf_verifier_env *env, struct bpf_prog *prog) 3149 3089 { 3150 3090 return prog; 3151 3091 }
+137 -25
kernel/bpf/fixups.c
··· 183 183 data[i].seen = old_seen; 184 184 data[i].zext_dst = insn_has_def32(insn + i); 185 185 } 186 + 187 + /* 188 + * The indirect_target flag of the original instruction was moved to the last of the 189 + * new instructions by the above memmove and memset, but the indirect jump target is 190 + * actually the first instruction, so move it back. This also matches with the behavior 191 + * of bpf_insn_array_adjust(), which preserves xlated_off to point to the first new 192 + * instruction. 193 + */ 194 + if (data[off + cnt - 1].indirect_target) { 195 + data[off].indirect_target = 1; 196 + data[off + cnt - 1].indirect_target = 0; 197 + } 186 198 } 187 199 188 200 static void adjust_subprog_starts(struct bpf_verifier_env *env, u32 off, u32 len) ··· 244 232 } 245 233 } 246 234 247 - static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off, 248 - const struct bpf_insn *patch, u32 len) 235 + struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off, 236 + const struct bpf_insn *patch, u32 len) 249 237 { 250 238 struct bpf_prog *new_prog; 251 239 struct bpf_insn_aux_data *new_data = NULL; ··· 985 973 return 0; 986 974 } 987 975 988 - int bpf_jit_subprogs(struct bpf_verifier_env *env) 976 + static u32 *bpf_dup_subprog_starts(struct bpf_verifier_env *env) 977 + { 978 + u32 *starts = NULL; 979 + 980 + starts = kvmalloc_objs(u32, env->subprog_cnt, GFP_KERNEL_ACCOUNT); 981 + if (starts) { 982 + for (int i = 0; i < env->subprog_cnt; i++) 983 + starts[i] = env->subprog_info[i].start; 984 + } 985 + return starts; 986 + } 987 + 988 + static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_starts) 989 + { 990 + for (int i = 0; i < env->subprog_cnt; i++) 991 + env->subprog_info[i].start = orig_starts[i]; 992 + /* restore the start of fake 'exit' subprog as well */ 993 + env->subprog_info[env->subprog_cnt].start = env->prog->len; 994 + } 995 + 996 + struct bpf_insn_aux_data *bpf_dup_insn_aux_data(struct bpf_verifier_env *env) 997 + { 998 + size_t size; 999 + void *new_aux; 1000 + 1001 + size = array_size(sizeof(struct bpf_insn_aux_data), env->prog->len); 1002 + new_aux = __vmalloc(size, GFP_KERNEL_ACCOUNT); 1003 + if (new_aux) 1004 + memcpy(new_aux, env->insn_aux_data, size); 1005 + return new_aux; 1006 + } 1007 + 1008 + void bpf_restore_insn_aux_data(struct bpf_verifier_env *env, 1009 + struct bpf_insn_aux_data *orig_insn_aux) 1010 + { 1011 + /* the expanded elements are zero-filled, so no special handling is required */ 1012 + vfree(env->insn_aux_data); 1013 + env->insn_aux_data = orig_insn_aux; 1014 + } 1015 + 1016 + static int jit_subprogs(struct bpf_verifier_env *env) 989 1017 { 990 1018 struct bpf_prog *prog = env->prog, **func, *tmp; 991 1019 int i, j, subprog_start, subprog_end = 0, len, subprog; ··· 1033 981 struct bpf_insn *insn; 1034 982 void *old_bpf_func; 1035 983 int err, num_exentries; 1036 - int old_len, subprog_start_adjustment = 0; 1037 - 1038 - if (env->subprog_cnt <= 1) 1039 - return 0; 1040 984 1041 985 for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) { 1042 986 if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn)) ··· 1101 1053 goto out_free; 1102 1054 func[i]->is_func = 1; 1103 1055 func[i]->sleepable = prog->sleepable; 1056 + func[i]->blinded = prog->blinded; 1104 1057 func[i]->aux->func_idx = i; 1105 1058 /* Below members will be freed only at prog->aux */ 1106 1059 func[i]->aux->btf = prog->aux->btf; 1107 - func[i]->aux->subprog_start = subprog_start + subprog_start_adjustment; 1060 + func[i]->aux->subprog_start = subprog_start; 1108 1061 func[i]->aux->func_info = prog->aux->func_info; 1109 1062 func[i]->aux->func_info_cnt = prog->aux->func_info_cnt; 1110 1063 func[i]->aux->poke_tab = prog->aux->poke_tab; ··· 1162 1113 func[i]->aux->token = prog->aux->token; 1163 1114 if (!i) 1164 1115 func[i]->aux->exception_boundary = env->seen_exception; 1165 - 1166 - /* 1167 - * To properly pass the absolute subprog start to jit 1168 - * all instruction adjustments should be accumulated 1169 - */ 1170 - old_len = func[i]->len; 1171 - func[i] = bpf_int_jit_compile(func[i]); 1172 - subprog_start_adjustment += func[i]->len - old_len; 1173 - 1116 + func[i] = bpf_int_jit_compile(env, func[i]); 1174 1117 if (!func[i]->jited) { 1175 1118 err = -ENOTSUPP; 1176 1119 goto out_free; ··· 1206 1165 } 1207 1166 for (i = 0; i < env->subprog_cnt; i++) { 1208 1167 old_bpf_func = func[i]->bpf_func; 1209 - tmp = bpf_int_jit_compile(func[i]); 1168 + tmp = bpf_int_jit_compile(env, func[i]); 1210 1169 if (tmp != func[i] || func[i]->bpf_func != old_bpf_func) { 1211 1170 verbose(env, "JIT doesn't support bpf-to-bpf calls\n"); 1212 1171 err = -ENOTSUPP; ··· 1288 1247 } 1289 1248 kfree(func); 1290 1249 out_undo_insn: 1250 + bpf_prog_jit_attempt_done(prog); 1251 + return err; 1252 + } 1253 + 1254 + int bpf_jit_subprogs(struct bpf_verifier_env *env) 1255 + { 1256 + int err, i; 1257 + bool blinded = false; 1258 + struct bpf_insn *insn; 1259 + struct bpf_prog *prog, *orig_prog; 1260 + struct bpf_insn_aux_data *orig_insn_aux; 1261 + u32 *orig_subprog_starts; 1262 + 1263 + if (env->subprog_cnt <= 1) 1264 + return 0; 1265 + 1266 + prog = orig_prog = env->prog; 1267 + if (bpf_prog_need_blind(prog)) { 1268 + orig_insn_aux = bpf_dup_insn_aux_data(env); 1269 + if (!orig_insn_aux) { 1270 + err = -ENOMEM; 1271 + goto out_cleanup; 1272 + } 1273 + orig_subprog_starts = bpf_dup_subprog_starts(env); 1274 + if (!orig_subprog_starts) { 1275 + vfree(orig_insn_aux); 1276 + err = -ENOMEM; 1277 + goto out_cleanup; 1278 + } 1279 + prog = bpf_jit_blind_constants(env, prog); 1280 + if (IS_ERR(prog)) { 1281 + err = -ENOMEM; 1282 + prog = orig_prog; 1283 + goto out_restore; 1284 + } 1285 + blinded = true; 1286 + } 1287 + 1288 + err = jit_subprogs(env); 1289 + if (err) 1290 + goto out_jit_err; 1291 + 1292 + if (blinded) { 1293 + bpf_jit_prog_release_other(prog, orig_prog); 1294 + kvfree(orig_subprog_starts); 1295 + vfree(orig_insn_aux); 1296 + } 1297 + 1298 + return 0; 1299 + 1300 + out_jit_err: 1301 + if (blinded) { 1302 + bpf_jit_prog_release_other(orig_prog, prog); 1303 + /* roll back to the clean original prog */ 1304 + prog = env->prog = orig_prog; 1305 + goto out_restore; 1306 + } else { 1307 + if (err != -EFAULT) { 1308 + /* 1309 + * We will fall back to interpreter mode when err is not -EFAULT, before 1310 + * that, insn->off and insn->imm should be restored to their original 1311 + * values since they were modified by jit_subprogs. 1312 + */ 1313 + for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) { 1314 + if (!bpf_pseudo_call(insn)) 1315 + continue; 1316 + insn->off = 0; 1317 + insn->imm = env->insn_aux_data[i].call_imm; 1318 + } 1319 + } 1320 + goto out_cleanup; 1321 + } 1322 + 1323 + out_restore: 1324 + bpf_restore_subprog_starts(env, orig_subprog_starts); 1325 + bpf_restore_insn_aux_data(env, orig_insn_aux); 1326 + kvfree(orig_subprog_starts); 1327 + out_cleanup: 1291 1328 /* cleanup main prog to be interpreted */ 1292 1329 prog->jit_requested = 0; 1293 1330 prog->blinding_requested = 0; 1294 - for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) { 1295 - if (!bpf_pseudo_call(insn)) 1296 - continue; 1297 - insn->off = 0; 1298 - insn->imm = env->insn_aux_data[i].call_imm; 1299 - } 1300 - bpf_prog_jit_attempt_done(prog); 1301 1331 return err; 1302 1332 } 1303 1333
-4
kernel/bpf/syscall.c
··· 3083 3083 if (err < 0) 3084 3084 goto free_used_maps; 3085 3085 3086 - prog = bpf_prog_select_runtime(prog, &err); 3087 - if (err < 0) 3088 - goto free_used_maps; 3089 - 3090 3086 err = bpf_prog_mark_insn_arrays_ready(prog); 3091 3087 if (err < 0) 3092 3088 goto free_used_maps;
+15 -6
kernel/bpf/verifier.c
··· 3497 3497 return INSN_F_STACK_ACCESS | (spi << INSN_F_SPI_SHIFT) | frameno; 3498 3498 } 3499 3499 3500 + static void mark_indirect_target(struct bpf_verifier_env *env, int idx) 3501 + { 3502 + env->insn_aux_data[idx].indirect_target = true; 3503 + } 3504 + 3500 3505 #define LR_FRAMENO_BITS 3 3501 3506 #define LR_SPI_BITS 6 3502 3507 #define LR_ENTRY_BITS (LR_SPI_BITS + LR_FRAMENO_BITS + 1) ··· 17550 17545 } 17551 17546 17552 17547 for (i = 0; i < n - 1; i++) { 17548 + mark_indirect_target(env, env->gotox_tmp_buf->items[i]); 17553 17549 other_branch = push_stack(env, env->gotox_tmp_buf->items[i], 17554 17550 env->insn_idx, env->cur_state->speculative); 17555 17551 if (IS_ERR(other_branch)) 17556 17552 return PTR_ERR(other_branch); 17557 17553 } 17558 17554 env->insn_idx = env->gotox_tmp_buf->items[n-1]; 17555 + mark_indirect_target(env, env->insn_idx); 17559 17556 return INSN_IDX_UPDATED; 17560 17557 } 17561 17558 ··· 20162 20155 20163 20156 adjust_btf_func(env); 20164 20157 20158 + /* extension progs temporarily inherit the attach_type of their targets 20159 + for verification purposes, so set it back to zero before returning 20160 + */ 20161 + if (env->prog->type == BPF_PROG_TYPE_EXT) 20162 + env->prog->expected_attach_type = 0; 20163 + 20164 + env->prog = __bpf_prog_select_runtime(env, env->prog, &ret); 20165 + 20165 20166 err_release_maps: 20166 20167 if (ret) 20167 20168 release_insn_arrays(env); ··· 20180 20165 release_maps(env); 20181 20166 if (!env->prog->aux->used_btfs) 20182 20167 release_btfs(env); 20183 - 20184 - /* extension progs temporarily inherit the attach_type of their targets 20185 - for verification purposes, so set it back to zero before returning 20186 - */ 20187 - if (env->prog->type == BPF_PROG_TYPE_EXT) 20188 - env->prog->expected_attach_type = 0; 20189 20168 20190 20169 *prog = env->prog; 20191 20170