Linux kernel mirror (for testing) git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
kernel os linux
1
fork

Configure Feed

Select the types of activity you want to include in your feed.

bpf: Move constants blinding out of arch-specific JITs

During the JIT stage, constants blinding rewrites instructions but only
rewrites the private instruction copy of the JITed subprog, leaving the
global env->prog->insnsi and env->insn_aux_data untouched. This causes a
mismatch between subprog instructions and the global state, making it
difficult to use the global data in the JIT.

To avoid this mismatch, and given that all arch-specific JITs already
support constants blinding, move it to the generic verifier code, and
switch to rewrite the global env->prog->insnsi with the global states
adjusted, as other rewrites in the verifier do.

This removes the constants blinding calls in each JIT, which are largely
duplicated code across architectures.

Since constants blinding is only required for JIT, and there are two
JIT entry functions, jit_subprogs() for BPF programs with multiple
subprogs and bpf_prog_select_runtime() for programs with no subprogs,
move the constants blinding invocation into these two functions.

In the verifier path, bpf_patch_insn_data() is used to keep global
verifier auxiliary data in sync with patched instructions. A key
question is whether this global auxiliary data should be restored
on the failure path.

Besides instructions, bpf_patch_insn_data() adjusts:
- prog->aux->poke_tab
- env->insn_array_maps
- env->subprog_info
- env->insn_aux_data

For prog->aux->poke_tab, it is only used by JIT or only meaningful after
JIT succeeds, so it does not need to be restored on the failure path.

For env->insn_array_maps, when JIT fails, programs using insn arrays
are rejected by bpf_insn_array_ready() due to missing JIT addresses.
Hence, env->insn_array_maps is only meaningful for JIT and does not need
to be restored.

For subprog_info, if jit_subprogs fails and CONFIG_BPF_JIT_ALWAYS_ON
is not enabled, kernel falls back to interpreter. In this case,
env->subprog_info is used to determine subprogram stack depth. So it
must be restored on failure.

For env->insn_aux_data, it is freed by clear_insn_aux_data() at the
end of bpf_check(). Before freeing, clear_insn_aux_data() loops over
env->insn_aux_data to release jump targets recorded in it. The loop
uses env->prog->len as the array length, but this length no longer
matches the actual size of the adjusted env->insn_aux_data array after
constants blinding.

To address it, a simple approach is to keep insn_aux_data as adjusted
after failure, since it will be freed shortly, and record its actual size
for the loop in clear_insn_aux_data(). But since clear_insn_aux_data()
uses the same index to loop over both env->prog->insnsi and env->insn_aux_data,
this approach results in incorrect index for the insnsi array. So an
alternative approach is adopted: clone the original env->insn_aux_data
before blinding and restore it after failure, similar to env->prog.

For classic BPF programs, constants blinding works as before since it
is still invoked from bpf_prog_select_runtime().

Reviewed-by: Anton Protopopov <a.s.protopopov@gmail.com> # v8
Reviewed-by: Hari Bathini <hbathini@linux.ibm.com> # powerpc jit
Reviewed-by: Pu Lehui <pulehui@huawei.com> # riscv jit
Acked-by: Hengqi Chen <hengqi.chen@gmail.com> # loongarch jit
Signed-off-by: Xu Kuohai <xukuohai@huawei.com>
Link: https://lore.kernel.org/r/20260416064341.151802-2-xukuohai@huaweicloud.com
Signed-off-by: Alexei Starovoitov <ast@kernel.org>

authored by

Xu Kuohai and committed by
Alexei Starovoitov
d3e94522 a2044665

+403 -478
+14 -25
arch/arc/net/bpf_jit_core.c
··· 79 79 * The JIT pertinent context that is used by different functions. 80 80 * 81 81 * prog: The current eBPF program being handled. 82 - * orig_prog: The original eBPF program before any possible change. 83 82 * jit: The JIT buffer and its length. 84 83 * bpf_header: The JITed program header. "jit.buf" points inside it. 85 84 * emit: If set, opcodes are written to memory; else, a dry-run. ··· 93 94 * need_extra_pass: A forecast if an "extra_pass" will occur. 94 95 * is_extra_pass: Indicates if the current pass is an extra pass. 95 96 * user_bpf_prog: True, if VM opcodes come from a real program. 96 - * blinded: True if "constant blinding" step returned a new "prog". 97 97 * success: Indicates if the whole JIT went OK. 98 98 */ 99 99 struct jit_context { 100 100 struct bpf_prog *prog; 101 - struct bpf_prog *orig_prog; 102 101 struct jit_buffer jit; 103 102 struct bpf_binary_header *bpf_header; 104 103 bool emit; ··· 111 114 bool need_extra_pass; 112 115 bool is_extra_pass; 113 116 bool user_bpf_prog; 114 - bool blinded; 115 117 bool success; 116 118 }; 117 119 ··· 157 161 { 158 162 memset(ctx, 0, sizeof(*ctx)); 159 163 160 - ctx->orig_prog = prog; 161 - 162 - /* If constant blinding was requested but failed, scram. */ 163 - ctx->prog = bpf_jit_blind_constants(prog); 164 - if (IS_ERR(ctx->prog)) 165 - return PTR_ERR(ctx->prog); 166 - ctx->blinded = (ctx->prog != ctx->orig_prog); 164 + ctx->prog = prog; 167 165 168 166 /* If the verifier doesn't zero-extend, then we have to do it. */ 169 167 ctx->do_zext = !ctx->prog->aux->verifier_zext; ··· 204 214 */ 205 215 static void jit_ctx_cleanup(struct jit_context *ctx) 206 216 { 207 - if (ctx->blinded) { 208 - /* if all went well, release the orig_prog. */ 209 - if (ctx->success) 210 - bpf_jit_prog_release_other(ctx->prog, ctx->orig_prog); 211 - else 212 - bpf_jit_prog_release_other(ctx->orig_prog, ctx->prog); 213 - } 214 - 215 217 maybe_free(ctx, (void **)&ctx->bpf2insn); 216 218 maybe_free(ctx, (void **)&ctx->jit_data); 217 219 ··· 211 229 ctx->bpf2insn_valid = false; 212 230 213 231 /* Freeing "bpf_header" is enough. "jit.buf" is a sub-array of it. */ 214 - if (!ctx->success && ctx->bpf_header) { 215 - bpf_jit_binary_free(ctx->bpf_header); 216 - ctx->bpf_header = NULL; 217 - ctx->jit.buf = NULL; 218 - ctx->jit.index = 0; 219 - ctx->jit.len = 0; 232 + if (!ctx->success) { 233 + if (ctx->bpf_header) { 234 + bpf_jit_binary_free(ctx->bpf_header); 235 + ctx->bpf_header = NULL; 236 + ctx->jit.buf = NULL; 237 + ctx->jit.index = 0; 238 + ctx->jit.len = 0; 239 + } 240 + if (ctx->is_extra_pass) { 241 + ctx->prog->bpf_func = NULL; 242 + ctx->prog->jited = 0; 243 + ctx->prog->jited_len = 0; 244 + } 220 245 } 221 246 222 247 ctx->emit = false;
+7 -34
arch/arm/net/bpf_jit_32.c
··· 2144 2144 2145 2145 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 2146 2146 { 2147 - struct bpf_prog *tmp, *orig_prog = prog; 2148 2147 struct bpf_binary_header *header; 2149 - bool tmp_blinded = false; 2150 2148 struct jit_ctx ctx; 2151 2149 unsigned int tmp_idx; 2152 2150 unsigned int image_size; ··· 2154 2156 * the interpreter. 2155 2157 */ 2156 2158 if (!prog->jit_requested) 2157 - return orig_prog; 2158 - 2159 - /* If constant blinding was enabled and we failed during blinding 2160 - * then we must fall back to the interpreter. Otherwise, we save 2161 - * the new JITed code. 2162 - */ 2163 - tmp = bpf_jit_blind_constants(prog); 2164 - 2165 - if (IS_ERR(tmp)) 2166 - return orig_prog; 2167 - if (tmp != prog) { 2168 - tmp_blinded = true; 2169 - prog = tmp; 2170 - } 2159 + return prog; 2171 2160 2172 2161 memset(&ctx, 0, sizeof(ctx)); 2173 2162 ctx.prog = prog; ··· 2164 2179 * we must fall back to the interpreter 2165 2180 */ 2166 2181 ctx.offsets = kcalloc(prog->len, sizeof(int), GFP_KERNEL); 2167 - if (ctx.offsets == NULL) { 2168 - prog = orig_prog; 2169 - goto out; 2170 - } 2182 + if (ctx.offsets == NULL) 2183 + return prog; 2171 2184 2172 2185 /* 1) fake pass to find in the length of the JITed code, 2173 2186 * to compute ctx->offsets and other context variables ··· 2177 2194 * being successful in the second pass, so just fall back 2178 2195 * to the interpreter. 2179 2196 */ 2180 - if (build_body(&ctx)) { 2181 - prog = orig_prog; 2197 + if (build_body(&ctx)) 2182 2198 goto out_off; 2183 - } 2184 2199 2185 2200 tmp_idx = ctx.idx; 2186 2201 build_prologue(&ctx); ··· 2194 2213 ctx.idx += ctx.imm_count; 2195 2214 if (ctx.imm_count) { 2196 2215 ctx.imms = kcalloc(ctx.imm_count, sizeof(u32), GFP_KERNEL); 2197 - if (ctx.imms == NULL) { 2198 - prog = orig_prog; 2216 + if (ctx.imms == NULL) 2199 2217 goto out_off; 2200 - } 2201 2218 } 2202 2219 #else 2203 2220 /* there's nothing about the epilogue on ARMv7 */ ··· 2217 2238 /* Not able to allocate memory for the structure then 2218 2239 * we must fall back to the interpretation 2219 2240 */ 2220 - if (header == NULL) { 2221 - prog = orig_prog; 2241 + if (header == NULL) 2222 2242 goto out_imms; 2223 - } 2224 2243 2225 2244 /* 2.) Actual pass to generate final JIT code */ 2226 2245 ctx.target = (u32 *) image_ptr; ··· 2255 2278 #endif 2256 2279 out_off: 2257 2280 kfree(ctx.offsets); 2258 - out: 2259 - if (tmp_blinded) 2260 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 2261 - tmp : orig_prog); 2281 + 2262 2282 return prog; 2263 2283 2264 2284 out_free: 2265 2285 image_ptr = NULL; 2266 2286 bpf_jit_binary_free(header); 2267 - prog = orig_prog; 2268 2287 goto out_imms; 2269 2288 } 2270 2289
+20 -52
arch/arm64/net/bpf_jit_comp.c
··· 2003 2003 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 2004 2004 { 2005 2005 int image_size, prog_size, extable_size, extable_align, extable_offset; 2006 - struct bpf_prog *tmp, *orig_prog = prog; 2007 2006 struct bpf_binary_header *header; 2008 2007 struct bpf_binary_header *ro_header = NULL; 2009 2008 struct arm64_jit_data *jit_data; 2010 2009 void __percpu *priv_stack_ptr = NULL; 2011 2010 bool was_classic = bpf_prog_was_classic(prog); 2012 2011 int priv_stack_alloc_sz; 2013 - bool tmp_blinded = false; 2014 2012 bool extra_pass = false; 2015 2013 struct jit_ctx ctx; 2016 2014 u8 *image_ptr; ··· 2017 2019 int exentry_idx; 2018 2020 2019 2021 if (!prog->jit_requested) 2020 - return orig_prog; 2021 - 2022 - tmp = bpf_jit_blind_constants(prog); 2023 - /* If blinding was requested and we failed during blinding, 2024 - * we must fall back to the interpreter. 2025 - */ 2026 - if (IS_ERR(tmp)) 2027 - return orig_prog; 2028 - if (tmp != prog) { 2029 - tmp_blinded = true; 2030 - prog = tmp; 2031 - } 2022 + return prog; 2032 2023 2033 2024 jit_data = prog->aux->jit_data; 2034 2025 if (!jit_data) { 2035 2026 jit_data = kzalloc_obj(*jit_data); 2036 - if (!jit_data) { 2037 - prog = orig_prog; 2038 - goto out; 2039 - } 2027 + if (!jit_data) 2028 + return prog; 2040 2029 prog->aux->jit_data = jit_data; 2041 2030 } 2042 2031 priv_stack_ptr = prog->aux->priv_stack_ptr; ··· 2035 2050 priv_stack_alloc_sz = round_up(prog->aux->stack_depth, 16) + 2036 2051 2 * PRIV_STACK_GUARD_SZ; 2037 2052 priv_stack_ptr = __alloc_percpu_gfp(priv_stack_alloc_sz, 16, GFP_KERNEL); 2038 - if (!priv_stack_ptr) { 2039 - prog = orig_prog; 2053 + if (!priv_stack_ptr) 2040 2054 goto out_priv_stack; 2041 - } 2042 2055 2043 2056 priv_stack_init_guard(priv_stack_ptr, priv_stack_alloc_sz); 2044 2057 prog->aux->priv_stack_ptr = priv_stack_ptr; ··· 2056 2073 ctx.prog = prog; 2057 2074 2058 2075 ctx.offset = kvzalloc_objs(int, prog->len + 1); 2059 - if (ctx.offset == NULL) { 2060 - prog = orig_prog; 2076 + if (ctx.offset == NULL) 2061 2077 goto out_off; 2062 - } 2063 2078 2064 2079 ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena); 2065 2080 ctx.arena_vm_start = bpf_arena_get_kern_vm_start(prog->aux->arena); ··· 2070 2089 * BPF line info needs ctx->offset[i] to be the offset of 2071 2090 * instruction[i] in jited image, so build prologue first. 2072 2091 */ 2073 - if (build_prologue(&ctx, was_classic)) { 2074 - prog = orig_prog; 2092 + if (build_prologue(&ctx, was_classic)) 2075 2093 goto out_off; 2076 - } 2077 2094 2078 - if (build_body(&ctx, extra_pass)) { 2079 - prog = orig_prog; 2095 + if (build_body(&ctx, extra_pass)) 2080 2096 goto out_off; 2081 - } 2082 2097 2083 2098 ctx.epilogue_offset = ctx.idx; 2084 2099 build_epilogue(&ctx, was_classic); ··· 2092 2115 ro_header = bpf_jit_binary_pack_alloc(image_size, &ro_image_ptr, 2093 2116 sizeof(u64), &header, &image_ptr, 2094 2117 jit_fill_hole); 2095 - if (!ro_header) { 2096 - prog = orig_prog; 2118 + if (!ro_header) 2097 2119 goto out_off; 2098 - } 2099 2120 2100 2121 /* Pass 2: Determine jited position and result for each instruction */ 2101 2122 ··· 2121 2146 /* Dont write body instructions to memory for now */ 2122 2147 ctx.write = false; 2123 2148 2124 - if (build_body(&ctx, extra_pass)) { 2125 - prog = orig_prog; 2149 + if (build_body(&ctx, extra_pass)) 2126 2150 goto out_free_hdr; 2127 - } 2128 2151 2129 2152 ctx.epilogue_offset = ctx.idx; 2130 2153 ctx.exentry_idx = exentry_idx; ··· 2131 2158 2132 2159 /* Pass 3: Adjust jump offset and write final image */ 2133 2160 if (build_body(&ctx, extra_pass) || 2134 - WARN_ON_ONCE(ctx.idx != ctx.epilogue_offset)) { 2135 - prog = orig_prog; 2161 + WARN_ON_ONCE(ctx.idx != ctx.epilogue_offset)) 2136 2162 goto out_free_hdr; 2137 - } 2138 2163 2139 2164 build_epilogue(&ctx, was_classic); 2140 2165 build_plt(&ctx); 2141 2166 2142 2167 /* Extra pass to validate JITed code. */ 2143 - if (validate_ctx(&ctx)) { 2144 - prog = orig_prog; 2168 + if (validate_ctx(&ctx)) 2145 2169 goto out_free_hdr; 2146 - } 2147 2170 2148 2171 /* update the real prog size */ 2149 2172 prog_size = sizeof(u32) * ctx.idx; ··· 2156 2187 if (extra_pass && ctx.idx > jit_data->ctx.idx) { 2157 2188 pr_err_once("multi-func JIT bug %d > %d\n", 2158 2189 ctx.idx, jit_data->ctx.idx); 2159 - prog->bpf_func = NULL; 2160 - prog->jited = 0; 2161 - prog->jited_len = 0; 2162 2190 goto out_free_hdr; 2163 2191 } 2164 2192 if (WARN_ON(bpf_jit_binary_pack_finalize(ro_header, header))) { 2165 - /* ro_header has been freed */ 2193 + /* ro_header and header has been freed */ 2166 2194 ro_header = NULL; 2167 - prog = orig_prog; 2168 - goto out_off; 2195 + header = NULL; 2196 + goto out_free_hdr; 2169 2197 } 2170 2198 } else { 2171 2199 jit_data->ctx = ctx; ··· 2199 2233 kfree(jit_data); 2200 2234 prog->aux->jit_data = NULL; 2201 2235 } 2202 - out: 2203 - if (tmp_blinded) 2204 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 2205 - tmp : orig_prog); 2236 + 2206 2237 return prog; 2207 2238 2208 2239 out_free_hdr: 2240 + if (extra_pass) { 2241 + prog->bpf_func = NULL; 2242 + prog->jited = 0; 2243 + prog->jited_len = 0; 2244 + } 2209 2245 if (header) { 2210 2246 bpf_arch_text_copy(&ro_header->size, &header->size, 2211 2247 sizeof(header->size));
+17 -42
arch/loongarch/net/bpf_jit.c
··· 1922 1922 1923 1923 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 1924 1924 { 1925 - bool tmp_blinded = false, extra_pass = false; 1925 + bool extra_pass = false; 1926 1926 u8 *image_ptr, *ro_image_ptr; 1927 1927 int image_size, prog_size, extable_size; 1928 1928 struct jit_ctx ctx; 1929 1929 struct jit_data *jit_data; 1930 1930 struct bpf_binary_header *header; 1931 1931 struct bpf_binary_header *ro_header; 1932 - struct bpf_prog *tmp, *orig_prog = prog; 1933 1932 1934 1933 /* 1935 1934 * If BPF JIT was not enabled then we must fall back to 1936 1935 * the interpreter. 1937 1936 */ 1938 1937 if (!prog->jit_requested) 1939 - return orig_prog; 1940 - 1941 - tmp = bpf_jit_blind_constants(prog); 1942 - /* 1943 - * If blinding was requested and we failed during blinding, 1944 - * we must fall back to the interpreter. Otherwise, we save 1945 - * the new JITed code. 1946 - */ 1947 - if (IS_ERR(tmp)) 1948 - return orig_prog; 1949 - 1950 - if (tmp != prog) { 1951 - tmp_blinded = true; 1952 - prog = tmp; 1953 - } 1938 + return prog; 1954 1939 1955 1940 jit_data = prog->aux->jit_data; 1956 1941 if (!jit_data) { 1957 1942 jit_data = kzalloc_obj(*jit_data); 1958 - if (!jit_data) { 1959 - prog = orig_prog; 1960 - goto out; 1961 - } 1943 + if (!jit_data) 1944 + return prog; 1962 1945 prog->aux->jit_data = jit_data; 1963 1946 } 1964 1947 if (jit_data->ctx.offset) { ··· 1961 1978 ctx.user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena); 1962 1979 1963 1980 ctx.offset = kvcalloc(prog->len + 1, sizeof(u32), GFP_KERNEL); 1964 - if (ctx.offset == NULL) { 1965 - prog = orig_prog; 1981 + if (ctx.offset == NULL) 1966 1982 goto out_offset; 1967 - } 1968 1983 1969 1984 /* 1. Initial fake pass to compute ctx->idx and set ctx->flags */ 1970 1985 build_prologue(&ctx); 1971 - if (build_body(&ctx, extra_pass)) { 1972 - prog = orig_prog; 1986 + if (build_body(&ctx, extra_pass)) 1973 1987 goto out_offset; 1974 - } 1975 1988 ctx.epilogue_offset = ctx.idx; 1976 1989 build_epilogue(&ctx); 1977 1990 ··· 1983 2004 /* Now we know the size of the structure to make */ 1984 2005 ro_header = bpf_jit_binary_pack_alloc(image_size, &ro_image_ptr, sizeof(u32), 1985 2006 &header, &image_ptr, jit_fill_hole); 1986 - if (!ro_header) { 1987 - prog = orig_prog; 2007 + if (!ro_header) 1988 2008 goto out_offset; 1989 - } 1990 2009 1991 2010 /* 2. Now, the actual pass to generate final JIT code */ 1992 2011 /* ··· 2004 2027 ctx.num_exentries = 0; 2005 2028 2006 2029 build_prologue(&ctx); 2007 - if (build_body(&ctx, extra_pass)) { 2008 - prog = orig_prog; 2030 + if (build_body(&ctx, extra_pass)) 2009 2031 goto out_free; 2010 - } 2011 2032 build_epilogue(&ctx); 2012 2033 2013 2034 /* 3. Extra pass to validate JITed code */ 2014 - if (validate_ctx(&ctx)) { 2015 - prog = orig_prog; 2035 + if (validate_ctx(&ctx)) 2016 2036 goto out_free; 2017 - } 2018 2037 2019 2038 /* And we're done */ 2020 2039 if (bpf_jit_enable > 1) ··· 2023 2050 goto out_free; 2024 2051 } 2025 2052 if (WARN_ON(bpf_jit_binary_pack_finalize(ro_header, header))) { 2026 - /* ro_header has been freed */ 2053 + /* ro_header and header have been freed */ 2027 2054 ro_header = NULL; 2028 - prog = orig_prog; 2055 + header = NULL; 2029 2056 goto out_free; 2030 2057 } 2031 2058 /* ··· 2057 2084 prog->aux->jit_data = NULL; 2058 2085 } 2059 2086 2060 - out: 2061 - if (tmp_blinded) 2062 - bpf_jit_prog_release_other(prog, prog == orig_prog ? tmp : orig_prog); 2063 - 2064 2087 return prog; 2065 2088 2066 2089 out_free: 2090 + if (extra_pass) { 2091 + prog->bpf_func = NULL; 2092 + prog->jited = 0; 2093 + prog->jited_len = 0; 2094 + } 2095 + 2067 2096 if (header) { 2068 2097 bpf_arch_text_copy(&ro_header->size, &header->size, sizeof(header->size)); 2069 2098 bpf_jit_binary_pack_free(ro_header, header);
+1 -19
arch/mips/net/bpf_jit_comp.c
··· 911 911 912 912 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 913 913 { 914 - struct bpf_prog *tmp, *orig_prog = prog; 915 914 struct bpf_binary_header *header = NULL; 916 915 struct jit_context ctx; 917 - bool tmp_blinded = false; 918 916 unsigned int tmp_idx; 919 917 unsigned int image_size; 920 918 u8 *image_ptr; ··· 923 925 * the interpreter. 924 926 */ 925 927 if (!prog->jit_requested) 926 - return orig_prog; 927 - /* 928 - * If constant blinding was enabled and we failed during blinding 929 - * then we must fall back to the interpreter. Otherwise, we save 930 - * the new JITed code. 931 - */ 932 - tmp = bpf_jit_blind_constants(prog); 933 - if (IS_ERR(tmp)) 934 - return orig_prog; 935 - if (tmp != prog) { 936 - tmp_blinded = true; 937 - prog = tmp; 938 - } 928 + return prog; 939 929 940 930 memset(&ctx, 0, sizeof(ctx)); 941 931 ctx.program = prog; ··· 1011 1025 prog->jited_len = image_size; 1012 1026 1013 1027 out: 1014 - if (tmp_blinded) 1015 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 1016 - tmp : orig_prog); 1017 1028 kfree(ctx.descriptors); 1018 1029 return prog; 1019 1030 1020 1031 out_err: 1021 - prog = orig_prog; 1022 1032 if (header) 1023 1033 bpf_jit_binary_free(header); 1024 1034 goto out;
+29 -44
arch/parisc/net/bpf_jit_core.c
··· 44 44 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 45 45 { 46 46 unsigned int prog_size = 0, extable_size = 0; 47 - bool tmp_blinded = false, extra_pass = false; 48 - struct bpf_prog *tmp, *orig_prog = prog; 47 + bool extra_pass = false; 49 48 int pass = 0, prev_ninsns = 0, prologue_len, i; 50 49 struct hppa_jit_data *jit_data; 51 50 struct hppa_jit_context *ctx; 52 51 53 52 if (!prog->jit_requested) 54 - return orig_prog; 55 - 56 - tmp = bpf_jit_blind_constants(prog); 57 - if (IS_ERR(tmp)) 58 - return orig_prog; 59 - if (tmp != prog) { 60 - tmp_blinded = true; 61 - prog = tmp; 62 - } 53 + return prog; 63 54 64 55 jit_data = prog->aux->jit_data; 65 56 if (!jit_data) { 66 57 jit_data = kzalloc_obj(*jit_data); 67 - if (!jit_data) { 68 - prog = orig_prog; 69 - goto out; 70 - } 58 + if (!jit_data) 59 + return prog; 71 60 prog->aux->jit_data = jit_data; 72 61 } 73 62 ··· 70 81 71 82 ctx->prog = prog; 72 83 ctx->offset = kzalloc_objs(int, prog->len); 73 - if (!ctx->offset) { 74 - prog = orig_prog; 75 - goto out_offset; 76 - } 84 + if (!ctx->offset) 85 + goto out_err; 77 86 for (i = 0; i < prog->len; i++) { 78 87 prev_ninsns += 20; 79 88 ctx->offset[i] = prev_ninsns; ··· 80 93 for (i = 0; i < NR_JIT_ITERATIONS; i++) { 81 94 pass++; 82 95 ctx->ninsns = 0; 83 - if (build_body(ctx, extra_pass, ctx->offset)) { 84 - prog = orig_prog; 85 - goto out_offset; 86 - } 96 + if (build_body(ctx, extra_pass, ctx->offset)) 97 + goto out_err; 87 98 ctx->body_len = ctx->ninsns; 88 99 bpf_jit_build_prologue(ctx); 89 100 ctx->prologue_len = ctx->ninsns - ctx->body_len; ··· 101 116 &jit_data->image, 102 117 sizeof(long), 103 118 bpf_fill_ill_insns); 104 - if (!jit_data->header) { 105 - prog = orig_prog; 106 - goto out_offset; 107 - } 119 + if (!jit_data->header) 120 + goto out_err; 108 121 109 122 ctx->insns = (u32 *)jit_data->image; 110 123 /* ··· 117 134 pr_err("bpf-jit: image did not converge in <%d passes!\n", i); 118 135 if (jit_data->header) 119 136 bpf_jit_binary_free(jit_data->header); 120 - prog = orig_prog; 121 - goto out_offset; 137 + goto out_err; 122 138 } 123 139 124 140 if (extable_size) ··· 130 148 bpf_jit_build_prologue(ctx); 131 149 if (build_body(ctx, extra_pass, NULL)) { 132 150 bpf_jit_binary_free(jit_data->header); 133 - prog = orig_prog; 134 - goto out_offset; 151 + goto out_err; 135 152 } 136 153 bpf_jit_build_epilogue(ctx); 137 154 ··· 141 160 { extern int machine_restart(char *); machine_restart(""); } 142 161 } 143 162 163 + if (!prog->is_func || extra_pass) { 164 + if (bpf_jit_binary_lock_ro(jit_data->header)) { 165 + bpf_jit_binary_free(jit_data->header); 166 + goto out_err; 167 + } 168 + bpf_flush_icache(jit_data->header, ctx->insns + ctx->ninsns); 169 + } 170 + 144 171 prog->bpf_func = (void *)ctx->insns; 145 172 prog->jited = 1; 146 173 prog->jited_len = prog_size; 147 174 148 - bpf_flush_icache(jit_data->header, ctx->insns + ctx->ninsns); 149 - 150 175 if (!prog->is_func || extra_pass) { 151 - if (bpf_jit_binary_lock_ro(jit_data->header)) { 152 - bpf_jit_binary_free(jit_data->header); 153 - prog->bpf_func = NULL; 154 - prog->jited = 0; 155 - prog->jited_len = 0; 156 - goto out_offset; 157 - } 158 176 prologue_len = ctx->epilogue_offset - ctx->body_len; 159 177 for (i = 0; i < prog->len; i++) 160 178 ctx->offset[i] += prologue_len; ··· 163 183 kfree(jit_data); 164 184 prog->aux->jit_data = NULL; 165 185 } 166 - out: 186 + 167 187 if (HPPA_JIT_REBOOT) 168 188 { extern int machine_restart(char *); machine_restart(""); } 169 189 170 - if (tmp_blinded) 171 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 172 - tmp : orig_prog); 173 190 return prog; 191 + 192 + out_err: 193 + if (extra_pass) { 194 + prog->bpf_func = NULL; 195 + prog->jited = 0; 196 + prog->jited_len = 0; 197 + } 198 + goto out_offset; 174 199 } 175 200 176 201 u64 hppa_div64(u64 div, u64 divisor)
+26 -46
arch/powerpc/net/bpf_jit_comp.c
··· 177 177 void __percpu *priv_stack_ptr = NULL; 178 178 struct bpf_binary_header *fhdr = NULL; 179 179 struct bpf_binary_header *hdr = NULL; 180 - struct bpf_prog *org_fp = fp; 181 - struct bpf_prog *tmp_fp = NULL; 182 - bool bpf_blinded = false; 183 180 bool extra_pass = false; 184 181 u8 *fimage = NULL; 185 182 u32 *fcode_base = NULL; ··· 184 187 u32 fixup_len; 185 188 186 189 if (!fp->jit_requested) 187 - return org_fp; 188 - 189 - tmp_fp = bpf_jit_blind_constants(org_fp); 190 - if (IS_ERR(tmp_fp)) 191 - return org_fp; 192 - 193 - if (tmp_fp != org_fp) { 194 - bpf_blinded = true; 195 - fp = tmp_fp; 196 - } 190 + return fp; 197 191 198 192 jit_data = fp->aux->jit_data; 199 193 if (!jit_data) { 200 194 jit_data = kzalloc_obj(*jit_data); 201 - if (!jit_data) { 202 - fp = org_fp; 203 - goto out; 204 - } 195 + if (!jit_data) 196 + return fp; 205 197 fp->aux->jit_data = jit_data; 206 198 } 207 199 ··· 205 219 priv_stack_alloc_size = round_up(fp->aux->stack_depth, 16) + 206 220 2 * PRIV_STACK_GUARD_SZ; 207 221 priv_stack_ptr = __alloc_percpu_gfp(priv_stack_alloc_size, 16, GFP_KERNEL); 208 - if (!priv_stack_ptr) { 209 - fp = org_fp; 222 + if (!priv_stack_ptr) 210 223 goto out_priv_stack; 211 - } 212 224 213 225 priv_stack_init_guard(priv_stack_ptr, priv_stack_alloc_size); 214 226 fp->aux->priv_stack_ptr = priv_stack_ptr; ··· 233 249 } 234 250 235 251 addrs = kcalloc(flen + 1, sizeof(*addrs), GFP_KERNEL); 236 - if (addrs == NULL) { 237 - fp = org_fp; 238 - goto out_addrs; 239 - } 252 + if (addrs == NULL) 253 + goto out_err; 240 254 241 255 memset(&cgctx, 0, sizeof(struct codegen_context)); 242 256 bpf_jit_init_reg_mapping(&cgctx); ··· 261 279 } 262 280 263 281 /* Scouting faux-generate pass 0 */ 264 - if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) { 282 + if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) 265 283 /* We hit something illegal or unsupported. */ 266 - fp = org_fp; 267 - goto out_addrs; 268 - } 284 + goto out_err; 269 285 270 286 /* 271 287 * If we have seen a tail call, we need a second pass. ··· 274 294 */ 275 295 if (cgctx.seen & SEEN_TAILCALL || !is_offset_in_branch_range((long)cgctx.idx * 4)) { 276 296 cgctx.idx = 0; 277 - if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) { 278 - fp = org_fp; 279 - goto out_addrs; 280 - } 297 + if (bpf_jit_build_body(fp, NULL, NULL, &cgctx, addrs, 0, false)) 298 + goto out_err; 281 299 } 282 300 283 301 bpf_jit_realloc_regs(&cgctx); ··· 296 318 297 319 fhdr = bpf_jit_binary_pack_alloc(alloclen, &fimage, 4, &hdr, &image, 298 320 bpf_jit_fill_ill_insns); 299 - if (!fhdr) { 300 - fp = org_fp; 301 - goto out_addrs; 302 - } 321 + if (!fhdr) 322 + goto out_err; 303 323 304 324 if (extable_len) 305 325 fp->aux->extable = (void *)fimage + FUNCTION_DESCR_SIZE + proglen + fixup_len; ··· 316 340 extra_pass)) { 317 341 bpf_arch_text_copy(&fhdr->size, &hdr->size, sizeof(hdr->size)); 318 342 bpf_jit_binary_pack_free(fhdr, hdr); 319 - fp = org_fp; 320 - goto out_addrs; 343 + goto out_err; 321 344 } 322 345 bpf_jit_build_epilogue(code_base, &cgctx); 323 346 ··· 338 363 ((u64 *)image)[1] = local_paca->kernel_toc; 339 364 #endif 340 365 366 + if (!fp->is_func || extra_pass) { 367 + if (bpf_jit_binary_pack_finalize(fhdr, hdr)) 368 + goto out_err; 369 + } 370 + 341 371 fp->bpf_func = (void *)fimage; 342 372 fp->jited = 1; 343 373 fp->jited_len = cgctx.idx * 4 + FUNCTION_DESCR_SIZE; 344 374 345 375 if (!fp->is_func || extra_pass) { 346 - if (bpf_jit_binary_pack_finalize(fhdr, hdr)) { 347 - fp = org_fp; 348 - goto out_addrs; 349 - } 350 376 bpf_prog_fill_jited_linfo(fp, addrs); 351 377 /* 352 378 * On ABI V1, executable code starts after the function ··· 374 398 jit_data->hdr = hdr; 375 399 } 376 400 377 - out: 378 - if (bpf_blinded) 379 - bpf_jit_prog_release_other(fp, fp == org_fp ? tmp_fp : org_fp); 380 - 381 401 return fp; 402 + 403 + out_err: 404 + if (extra_pass) { 405 + fp->bpf_func = NULL; 406 + fp->jited = 0; 407 + fp->jited_len = 0; 408 + } 409 + goto out_addrs; 382 410 } 383 411 384 412 /*
+22 -39
arch/riscv/net/bpf_jit_core.c
··· 44 44 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 45 45 { 46 46 unsigned int prog_size = 0, extable_size = 0; 47 - bool tmp_blinded = false, extra_pass = false; 48 - struct bpf_prog *tmp, *orig_prog = prog; 47 + bool extra_pass = false; 49 48 int pass = 0, prev_ninsns = 0, i; 50 49 struct rv_jit_data *jit_data; 51 50 struct rv_jit_context *ctx; 52 51 53 52 if (!prog->jit_requested) 54 - return orig_prog; 55 - 56 - tmp = bpf_jit_blind_constants(prog); 57 - if (IS_ERR(tmp)) 58 - return orig_prog; 59 - if (tmp != prog) { 60 - tmp_blinded = true; 61 - prog = tmp; 62 - } 53 + return prog; 63 54 64 55 jit_data = prog->aux->jit_data; 65 56 if (!jit_data) { 66 57 jit_data = kzalloc_obj(*jit_data); 67 58 if (!jit_data) { 68 - prog = orig_prog; 69 - goto out; 59 + return prog; 70 60 } 71 61 prog->aux->jit_data = jit_data; 72 62 } ··· 73 83 ctx->user_vm_start = bpf_arena_get_user_vm_start(prog->aux->arena); 74 84 ctx->prog = prog; 75 85 ctx->offset = kzalloc_objs(int, prog->len); 76 - if (!ctx->offset) { 77 - prog = orig_prog; 86 + if (!ctx->offset) 78 87 goto out_offset; 79 - } 80 88 81 - if (build_body(ctx, extra_pass, NULL)) { 82 - prog = orig_prog; 89 + if (build_body(ctx, extra_pass, NULL)) 83 90 goto out_offset; 84 - } 85 91 86 92 for (i = 0; i < prog->len; i++) { 87 93 prev_ninsns += 32; ··· 91 105 bpf_jit_build_prologue(ctx, bpf_is_subprog(prog)); 92 106 ctx->prologue_len = ctx->ninsns; 93 107 94 - if (build_body(ctx, extra_pass, ctx->offset)) { 95 - prog = orig_prog; 108 + if (build_body(ctx, extra_pass, ctx->offset)) 96 109 goto out_offset; 97 - } 98 110 99 111 ctx->epilogue_offset = ctx->ninsns; 100 112 bpf_jit_build_epilogue(ctx); ··· 110 126 &jit_data->ro_image, sizeof(u32), 111 127 &jit_data->header, &jit_data->image, 112 128 bpf_fill_ill_insns); 113 - if (!jit_data->ro_header) { 114 - prog = orig_prog; 129 + if (!jit_data->ro_header) 115 130 goto out_offset; 116 - } 117 131 118 132 /* 119 133 * Use the image(RW) for writing the JITed instructions. But also save ··· 132 150 133 151 if (i == NR_JIT_ITERATIONS) { 134 152 pr_err("bpf-jit: image did not converge in <%d passes!\n", i); 135 - prog = orig_prog; 136 153 goto out_free_hdr; 137 154 } 138 155 ··· 144 163 ctx->nexentries = 0; 145 164 146 165 bpf_jit_build_prologue(ctx, bpf_is_subprog(prog)); 147 - if (build_body(ctx, extra_pass, NULL)) { 148 - prog = orig_prog; 166 + if (build_body(ctx, extra_pass, NULL)) 149 167 goto out_free_hdr; 150 - } 151 168 bpf_jit_build_epilogue(ctx); 152 169 153 170 if (bpf_jit_enable > 1) 154 171 bpf_jit_dump(prog->len, prog_size, pass, ctx->insns); 172 + 173 + if (!prog->is_func || extra_pass) { 174 + if (WARN_ON(bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->header))) { 175 + /* ro_header has been freed */ 176 + jit_data->ro_header = NULL; 177 + jit_data->header = NULL; 178 + goto out_free_hdr; 179 + } 180 + } 155 181 156 182 prog->bpf_func = (void *)ctx->ro_insns + cfi_get_offset(); 157 183 prog->jited = 1; 158 184 prog->jited_len = prog_size - cfi_get_offset(); 159 185 160 186 if (!prog->is_func || extra_pass) { 161 - if (WARN_ON(bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->header))) { 162 - /* ro_header has been freed */ 163 - jit_data->ro_header = NULL; 164 - prog = orig_prog; 165 - goto out_offset; 166 - } 167 187 for (i = 0; i < prog->len; i++) 168 188 ctx->offset[i] = ninsns_rvoff(ctx->offset[i]); 169 189 bpf_prog_fill_jited_linfo(prog, ctx->offset); ··· 173 191 kfree(jit_data); 174 192 prog->aux->jit_data = NULL; 175 193 } 176 - out: 177 194 178 - if (tmp_blinded) 179 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 180 - tmp : orig_prog); 181 195 return prog; 182 196 183 197 out_free_hdr: 198 + if (extra_pass) { 199 + prog->bpf_func = NULL; 200 + prog->jited = 0; 201 + prog->jited_len = 0; 202 + } 184 203 if (jit_data->header) { 185 204 bpf_arch_text_copy(&jit_data->ro_header->size, &jit_data->header->size, 186 205 sizeof(jit_data->header->size));
+20 -39
arch/s390/net/bpf_jit_comp.c
··· 2314 2314 */ 2315 2315 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) 2316 2316 { 2317 - struct bpf_prog *tmp, *orig_fp = fp; 2318 2317 struct bpf_binary_header *header; 2319 2318 struct s390_jit_data *jit_data; 2320 - bool tmp_blinded = false; 2321 2319 bool extra_pass = false; 2322 2320 struct bpf_jit jit; 2323 2321 int pass; 2324 2322 2325 2323 if (!fp->jit_requested) 2326 - return orig_fp; 2327 - 2328 - tmp = bpf_jit_blind_constants(fp); 2329 - /* 2330 - * If blinding was requested and we failed during blinding, 2331 - * we must fall back to the interpreter. 2332 - */ 2333 - if (IS_ERR(tmp)) 2334 - return orig_fp; 2335 - if (tmp != fp) { 2336 - tmp_blinded = true; 2337 - fp = tmp; 2338 - } 2324 + return fp; 2339 2325 2340 2326 jit_data = fp->aux->jit_data; 2341 2327 if (!jit_data) { 2342 2328 jit_data = kzalloc_obj(*jit_data); 2343 - if (!jit_data) { 2344 - fp = orig_fp; 2345 - goto out; 2346 - } 2329 + if (!jit_data) 2330 + return fp; 2347 2331 fp->aux->jit_data = jit_data; 2348 2332 } 2349 2333 if (jit_data->ctx.addrs) { ··· 2340 2356 2341 2357 memset(&jit, 0, sizeof(jit)); 2342 2358 jit.addrs = kvcalloc(fp->len + 1, sizeof(*jit.addrs), GFP_KERNEL); 2343 - if (jit.addrs == NULL) { 2344 - fp = orig_fp; 2345 - goto free_addrs; 2346 - } 2359 + if (jit.addrs == NULL) 2360 + goto out_err; 2347 2361 /* 2348 2362 * Three initial passes: 2349 2363 * - 1/2: Determine clobbered registers 2350 2364 * - 3: Calculate program size and addrs array 2351 2365 */ 2352 2366 for (pass = 1; pass <= 3; pass++) { 2353 - if (bpf_jit_prog(&jit, fp, extra_pass)) { 2354 - fp = orig_fp; 2355 - goto free_addrs; 2356 - } 2367 + if (bpf_jit_prog(&jit, fp, extra_pass)) 2368 + goto out_err; 2357 2369 } 2358 2370 /* 2359 2371 * Final pass: Allocate and generate program 2360 2372 */ 2361 2373 header = bpf_jit_alloc(&jit, fp); 2362 - if (!header) { 2363 - fp = orig_fp; 2364 - goto free_addrs; 2365 - } 2374 + if (!header) 2375 + goto out_err; 2366 2376 skip_init_ctx: 2367 2377 if (bpf_jit_prog(&jit, fp, extra_pass)) { 2368 2378 bpf_jit_binary_free(header); 2369 - fp = orig_fp; 2370 - goto free_addrs; 2379 + goto out_err; 2371 2380 } 2372 2381 if (bpf_jit_enable > 1) { 2373 2382 bpf_jit_dump(fp->len, jit.size, pass, jit.prg_buf); ··· 2369 2392 if (!fp->is_func || extra_pass) { 2370 2393 if (bpf_jit_binary_lock_ro(header)) { 2371 2394 bpf_jit_binary_free(header); 2372 - fp = orig_fp; 2373 - goto free_addrs; 2395 + goto out_err; 2374 2396 } 2375 2397 } else { 2376 2398 jit_data->header = header; ··· 2387 2411 kfree(jit_data); 2388 2412 fp->aux->jit_data = NULL; 2389 2413 } 2390 - out: 2391 - if (tmp_blinded) 2392 - bpf_jit_prog_release_other(fp, fp == orig_fp ? 2393 - tmp : orig_fp); 2414 + 2394 2415 return fp; 2416 + 2417 + out_err: 2418 + if (extra_pass) { 2419 + fp->bpf_func = NULL; 2420 + fp->jited = 0; 2421 + fp->jited_len = 0; 2422 + } 2423 + goto free_addrs; 2395 2424 } 2396 2425 2397 2426 bool bpf_jit_supports_kfunc_call(void)
+21 -40
arch/sparc/net/bpf_jit_comp_64.c
··· 1479 1479 1480 1480 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 1481 1481 { 1482 - struct bpf_prog *tmp, *orig_prog = prog; 1483 1482 struct sparc64_jit_data *jit_data; 1484 1483 struct bpf_binary_header *header; 1485 1484 u32 prev_image_size, image_size; 1486 - bool tmp_blinded = false; 1487 1485 bool extra_pass = false; 1488 1486 struct jit_ctx ctx; 1489 1487 u8 *image_ptr; 1490 1488 int pass, i; 1491 1489 1492 1490 if (!prog->jit_requested) 1493 - return orig_prog; 1494 - 1495 - tmp = bpf_jit_blind_constants(prog); 1496 - /* If blinding was requested and we failed during blinding, 1497 - * we must fall back to the interpreter. 1498 - */ 1499 - if (IS_ERR(tmp)) 1500 - return orig_prog; 1501 - if (tmp != prog) { 1502 - tmp_blinded = true; 1503 - prog = tmp; 1504 - } 1491 + return prog; 1505 1492 1506 1493 jit_data = prog->aux->jit_data; 1507 1494 if (!jit_data) { 1508 1495 jit_data = kzalloc_obj(*jit_data); 1509 - if (!jit_data) { 1510 - prog = orig_prog; 1511 - goto out; 1512 - } 1496 + if (!jit_data) 1497 + return prog; 1513 1498 prog->aux->jit_data = jit_data; 1514 1499 } 1515 1500 if (jit_data->ctx.offset) { ··· 1512 1527 ctx.prog = prog; 1513 1528 1514 1529 ctx.offset = kmalloc_array(prog->len, sizeof(unsigned int), GFP_KERNEL); 1515 - if (ctx.offset == NULL) { 1516 - prog = orig_prog; 1517 - goto out_off; 1518 - } 1530 + if (ctx.offset == NULL) 1531 + goto out_err; 1519 1532 1520 1533 /* Longest sequence emitted is for bswap32, 12 instructions. Pre-cook 1521 1534 * the offset array so that we converge faster. ··· 1526 1543 ctx.idx = 0; 1527 1544 1528 1545 build_prologue(&ctx); 1529 - if (build_body(&ctx)) { 1530 - prog = orig_prog; 1531 - goto out_off; 1532 - } 1546 + if (build_body(&ctx)) 1547 + goto out_err; 1533 1548 build_epilogue(&ctx); 1534 1549 1535 1550 if (bpf_jit_enable > 1) ··· 1550 1569 image_size = sizeof(u32) * ctx.idx; 1551 1570 header = bpf_jit_binary_alloc(image_size, &image_ptr, 1552 1571 sizeof(u32), jit_fill_hole); 1553 - if (header == NULL) { 1554 - prog = orig_prog; 1555 - goto out_off; 1556 - } 1572 + if (header == NULL) 1573 + goto out_err; 1557 1574 1558 1575 ctx.image = (u32 *)image_ptr; 1559 1576 skip_init_ctx: ··· 1561 1582 1562 1583 if (build_body(&ctx)) { 1563 1584 bpf_jit_binary_free(header); 1564 - prog = orig_prog; 1565 - goto out_off; 1585 + goto out_err; 1566 1586 } 1567 1587 1568 1588 build_epilogue(&ctx); ··· 1570 1592 pr_err("bpf_jit: Failed to converge, prev_size=%u size=%d\n", 1571 1593 prev_image_size, ctx.idx * 4); 1572 1594 bpf_jit_binary_free(header); 1573 - prog = orig_prog; 1574 - goto out_off; 1595 + goto out_err; 1575 1596 } 1576 1597 1577 1598 if (bpf_jit_enable > 1) ··· 1581 1604 if (!prog->is_func || extra_pass) { 1582 1605 if (bpf_jit_binary_lock_ro(header)) { 1583 1606 bpf_jit_binary_free(header); 1584 - prog = orig_prog; 1585 - goto out_off; 1607 + goto out_err; 1586 1608 } 1587 1609 } else { 1588 1610 jit_data->ctx = ctx; ··· 1600 1624 kfree(jit_data); 1601 1625 prog->aux->jit_data = NULL; 1602 1626 } 1603 - out: 1604 - if (tmp_blinded) 1605 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 1606 - tmp : orig_prog); 1627 + 1607 1628 return prog; 1629 + 1630 + out_err: 1631 + if (extra_pass) { 1632 + prog->bpf_func = NULL; 1633 + prog->jited = 0; 1634 + prog->jited_len = 0; 1635 + } 1636 + goto out_off; 1608 1637 }
+7 -36
arch/x86/net/bpf_jit_comp.c
··· 3717 3717 { 3718 3718 struct bpf_binary_header *rw_header = NULL; 3719 3719 struct bpf_binary_header *header = NULL; 3720 - struct bpf_prog *tmp, *orig_prog = prog; 3721 3720 void __percpu *priv_stack_ptr = NULL; 3722 3721 struct x64_jit_data *jit_data; 3723 3722 int priv_stack_alloc_sz; 3724 3723 int proglen, oldproglen = 0; 3725 3724 struct jit_context ctx = {}; 3726 - bool tmp_blinded = false; 3727 3725 bool extra_pass = false; 3728 3726 bool padding = false; 3729 3727 u8 *rw_image = NULL; ··· 3731 3733 int i; 3732 3734 3733 3735 if (!prog->jit_requested) 3734 - return orig_prog; 3735 - 3736 - tmp = bpf_jit_blind_constants(prog); 3737 - /* 3738 - * If blinding was requested and we failed during blinding, 3739 - * we must fall back to the interpreter. 3740 - */ 3741 - if (IS_ERR(tmp)) 3742 - return orig_prog; 3743 - if (tmp != prog) { 3744 - tmp_blinded = true; 3745 - prog = tmp; 3746 - } 3736 + return prog; 3747 3737 3748 3738 jit_data = prog->aux->jit_data; 3749 3739 if (!jit_data) { 3750 3740 jit_data = kzalloc_obj(*jit_data); 3751 - if (!jit_data) { 3752 - prog = orig_prog; 3753 - goto out; 3754 - } 3741 + if (!jit_data) 3742 + return prog; 3755 3743 prog->aux->jit_data = jit_data; 3756 3744 } 3757 3745 priv_stack_ptr = prog->aux->priv_stack_ptr; ··· 3749 3765 priv_stack_alloc_sz = round_up(prog->aux->stack_depth, 8) + 3750 3766 2 * PRIV_STACK_GUARD_SZ; 3751 3767 priv_stack_ptr = __alloc_percpu_gfp(priv_stack_alloc_sz, 8, GFP_KERNEL); 3752 - if (!priv_stack_ptr) { 3753 - prog = orig_prog; 3768 + if (!priv_stack_ptr) 3754 3769 goto out_priv_stack; 3755 - } 3756 3770 3757 3771 priv_stack_init_guard(priv_stack_ptr, priv_stack_alloc_sz); 3758 3772 prog->aux->priv_stack_ptr = priv_stack_ptr; ··· 3768 3786 goto skip_init_addrs; 3769 3787 } 3770 3788 addrs = kvmalloc_objs(*addrs, prog->len + 1); 3771 - if (!addrs) { 3772 - prog = orig_prog; 3789 + if (!addrs) 3773 3790 goto out_addrs; 3774 - } 3775 3791 3776 3792 /* 3777 3793 * Before first pass, make a rough estimation of addrs[] ··· 3800 3820 sizeof(rw_header->size)); 3801 3821 bpf_jit_binary_pack_free(header, rw_header); 3802 3822 } 3803 - /* Fall back to interpreter mode */ 3804 - prog = orig_prog; 3805 3823 if (extra_pass) { 3806 3824 prog->bpf_func = NULL; 3807 3825 prog->jited = 0; ··· 3830 3852 header = bpf_jit_binary_pack_alloc(roundup(proglen, align) + extable_size, 3831 3853 &image, align, &rw_header, &rw_image, 3832 3854 jit_fill_hole); 3833 - if (!header) { 3834 - prog = orig_prog; 3855 + if (!header) 3835 3856 goto out_addrs; 3836 - } 3837 3857 prog->aux->extable = (void *) image + roundup(proglen, align); 3838 3858 } 3839 3859 oldproglen = proglen; ··· 3884 3908 prog->bpf_func = (void *)image + cfi_get_offset(); 3885 3909 prog->jited = 1; 3886 3910 prog->jited_len = proglen - cfi_get_offset(); 3887 - } else { 3888 - prog = orig_prog; 3889 3911 } 3890 3912 3891 3913 if (!image || !prog->is_func || extra_pass) { ··· 3899 3925 kfree(jit_data); 3900 3926 prog->aux->jit_data = NULL; 3901 3927 } 3902 - out: 3903 - if (tmp_blinded) 3904 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 3905 - tmp : orig_prog); 3928 + 3906 3929 return prog; 3907 3930 } 3908 3931
+4 -29
arch/x86/net/bpf_jit_comp32.c
··· 2521 2521 struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) 2522 2522 { 2523 2523 struct bpf_binary_header *header = NULL; 2524 - struct bpf_prog *tmp, *orig_prog = prog; 2525 2524 int proglen, oldproglen = 0; 2526 2525 struct jit_context ctx = {}; 2527 - bool tmp_blinded = false; 2528 2526 u8 *image = NULL; 2529 2527 int *addrs; 2530 2528 int pass; 2531 2529 int i; 2532 2530 2533 2531 if (!prog->jit_requested) 2534 - return orig_prog; 2535 - 2536 - tmp = bpf_jit_blind_constants(prog); 2537 - /* 2538 - * If blinding was requested and we failed during blinding, 2539 - * we must fall back to the interpreter. 2540 - */ 2541 - if (IS_ERR(tmp)) 2542 - return orig_prog; 2543 - if (tmp != prog) { 2544 - tmp_blinded = true; 2545 - prog = tmp; 2546 - } 2532 + return prog; 2547 2533 2548 2534 addrs = kmalloc_objs(*addrs, prog->len); 2549 - if (!addrs) { 2550 - prog = orig_prog; 2551 - goto out; 2552 - } 2535 + if (!addrs) 2536 + return prog; 2553 2537 2554 2538 /* 2555 2539 * Before first pass, make a rough estimation of addrs[] ··· 2558 2574 image = NULL; 2559 2575 if (header) 2560 2576 bpf_jit_binary_free(header); 2561 - prog = orig_prog; 2562 2577 goto out_addrs; 2563 2578 } 2564 2579 if (image) { ··· 2571 2588 if (proglen == oldproglen) { 2572 2589 header = bpf_jit_binary_alloc(proglen, &image, 2573 2590 1, jit_fill_hole); 2574 - if (!header) { 2575 - prog = orig_prog; 2591 + if (!header) 2576 2592 goto out_addrs; 2577 - } 2578 2593 } 2579 2594 oldproglen = proglen; 2580 2595 cond_resched(); ··· 2585 2604 prog->bpf_func = (void *)image; 2586 2605 prog->jited = 1; 2587 2606 prog->jited_len = proglen; 2588 - } else { 2589 - prog = orig_prog; 2590 2607 } 2591 2608 2592 2609 out_addrs: 2593 2610 kfree(addrs); 2594 - out: 2595 - if (tmp_blinded) 2596 - bpf_jit_prog_release_other(prog, prog == orig_prog ? 2597 - tmp : orig_prog); 2598 2611 return prog; 2599 2612 } 2600 2613
+32 -1
include/linux/filter.h
··· 1184 1184 1185 1185 struct bpf_prog *bpf_patch_insn_single(struct bpf_prog *prog, u32 off, 1186 1186 const struct bpf_insn *patch, u32 len); 1187 + 1188 + #ifdef CONFIG_BPF_SYSCALL 1189 + struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off, 1190 + const struct bpf_insn *patch, u32 len); 1191 + #else 1192 + static inline struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off, 1193 + const struct bpf_insn *patch, u32 len) 1194 + { 1195 + return ERR_PTR(-ENOTSUPP); 1196 + } 1197 + #endif /* CONFIG_BPF_SYSCALL */ 1198 + 1187 1199 int bpf_remove_insns(struct bpf_prog *prog, u32 off, u32 cnt); 1188 1200 1189 1201 static inline bool xdp_return_frame_no_direct(void) ··· 1322 1310 1323 1311 const char *bpf_jit_get_prog_name(struct bpf_prog *prog); 1324 1312 1325 - struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *fp); 1313 + struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, struct bpf_prog *prog); 1326 1314 void bpf_jit_prog_release_other(struct bpf_prog *fp, struct bpf_prog *fp_other); 1315 + 1316 + static inline bool bpf_prog_need_blind(const struct bpf_prog *prog) 1317 + { 1318 + return prog->blinding_requested && !prog->blinded; 1319 + } 1327 1320 1328 1321 static inline void bpf_jit_dump(unsigned int flen, unsigned int proglen, 1329 1322 u32 pass, void *image) ··· 1468 1451 { 1469 1452 } 1470 1453 1454 + static inline bool bpf_prog_need_blind(const struct bpf_prog *prog) 1455 + { 1456 + return false; 1457 + } 1458 + 1459 + static inline 1460 + struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, struct bpf_prog *prog) 1461 + { 1462 + return prog; 1463 + } 1464 + 1465 + static inline void bpf_jit_prog_release_other(struct bpf_prog *fp, struct bpf_prog *fp_other) 1466 + { 1467 + } 1471 1468 #endif /* CONFIG_BPF_JIT */ 1472 1469 1473 1470 void bpf_prog_kallsyms_del_all(struct bpf_prog *fp);
+60 -9
kernel/bpf/core.c
··· 1508 1508 #endif 1509 1509 } 1510 1510 1511 - struct bpf_prog *bpf_jit_blind_constants(struct bpf_prog *prog) 1511 + /* 1512 + * Now this function is used only to blind the main prog and must be invoked only when 1513 + * bpf_prog_need_blind() returns true. 1514 + */ 1515 + struct bpf_prog *bpf_jit_blind_constants(struct bpf_verifier_env *env, struct bpf_prog *prog) 1512 1516 { 1513 1517 struct bpf_insn insn_buff[16], aux[2]; 1514 1518 struct bpf_prog *clone, *tmp; ··· 1520 1516 struct bpf_insn *insn; 1521 1517 int i, rewritten; 1522 1518 1523 - if (!prog->blinding_requested || prog->blinded) 1524 - return prog; 1519 + if (WARN_ON_ONCE(env && env->prog != prog)) 1520 + return ERR_PTR(-EINVAL); 1525 1521 1526 1522 clone = bpf_prog_clone_create(prog, GFP_USER); 1527 1523 if (!clone) 1528 1524 return ERR_PTR(-ENOMEM); 1525 + 1526 + /* make sure bpf_patch_insn_data() patches the correct prog */ 1527 + if (env) 1528 + env->prog = clone; 1529 1529 1530 1530 insn_cnt = clone->len; 1531 1531 insn = clone->insnsi; ··· 1558 1550 if (!rewritten) 1559 1551 continue; 1560 1552 1561 - tmp = bpf_patch_insn_single(clone, i, insn_buff, rewritten); 1562 - if (IS_ERR(tmp)) { 1553 + if (env) 1554 + tmp = bpf_patch_insn_data(env, i, insn_buff, rewritten); 1555 + else 1556 + tmp = bpf_patch_insn_single(clone, i, insn_buff, rewritten); 1557 + 1558 + if (IS_ERR_OR_NULL(tmp)) { 1559 + if (env) 1560 + /* restore the original prog */ 1561 + env->prog = prog; 1563 1562 /* Patching may have repointed aux->prog during 1564 1563 * realloc from the original one, so we need to 1565 1564 * fix it up here on error. 1566 1565 */ 1567 1566 bpf_jit_prog_release_other(prog, clone); 1568 - return tmp; 1567 + return IS_ERR(tmp) ? tmp : ERR_PTR(-ENOMEM); 1569 1568 } 1570 1569 1571 1570 clone = tmp; 1572 1571 insn_delta = rewritten - 1; 1573 1572 1574 - /* Instructions arrays must be updated using absolute xlated offsets */ 1575 - adjust_insn_arrays(clone, prog->aux->subprog_start + i, rewritten); 1573 + if (env) 1574 + env->prog = clone; 1575 + else 1576 + /* 1577 + * Instructions arrays must be updated using absolute xlated offsets. 1578 + * The arrays have already been adjusted by bpf_patch_insn_data() when 1579 + * env is not NULL. 1580 + */ 1581 + adjust_insn_arrays(clone, i, rewritten); 1576 1582 1577 1583 /* Walk new program and skip insns we just inserted. */ 1578 1584 insn = clone->insnsi + i + insn_delta; ··· 2555 2533 return select_interpreter; 2556 2534 } 2557 2535 2536 + static struct bpf_prog *bpf_prog_jit_compile(struct bpf_prog *prog) 2537 + { 2538 + #ifdef CONFIG_BPF_JIT 2539 + struct bpf_prog *orig_prog; 2540 + 2541 + if (!bpf_prog_need_blind(prog)) 2542 + return bpf_int_jit_compile(prog); 2543 + 2544 + orig_prog = prog; 2545 + prog = bpf_jit_blind_constants(NULL, prog); 2546 + /* 2547 + * If blinding was requested and we failed during blinding, we must fall 2548 + * back to the interpreter. 2549 + */ 2550 + if (IS_ERR(prog)) 2551 + return orig_prog; 2552 + 2553 + prog = bpf_int_jit_compile(prog); 2554 + if (prog->jited) { 2555 + bpf_jit_prog_release_other(prog, orig_prog); 2556 + return prog; 2557 + } 2558 + 2559 + bpf_jit_prog_release_other(orig_prog, prog); 2560 + prog = orig_prog; 2561 + #endif 2562 + return prog; 2563 + } 2564 + 2558 2565 /** 2559 2566 * bpf_prog_select_runtime - select exec runtime for BPF program 2560 2567 * @fp: bpf_prog populated with BPF program ··· 2623 2572 if (*err) 2624 2573 return fp; 2625 2574 2626 - fp = bpf_int_jit_compile(fp); 2575 + fp = bpf_prog_jit_compile(fp); 2627 2576 bpf_prog_jit_attempt_done(fp); 2628 2577 if (!fp->jited && jit_needed) { 2629 2578 *err = -ENOTSUPP;
+123 -23
kernel/bpf/fixups.c
··· 232 232 } 233 233 } 234 234 235 - static struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off, 236 - const struct bpf_insn *patch, u32 len) 235 + struct bpf_prog *bpf_patch_insn_data(struct bpf_verifier_env *env, u32 off, 236 + const struct bpf_insn *patch, u32 len) 237 237 { 238 238 struct bpf_prog *new_prog; 239 239 struct bpf_insn_aux_data *new_data = NULL; ··· 973 973 return 0; 974 974 } 975 975 976 - int bpf_jit_subprogs(struct bpf_verifier_env *env) 976 + static u32 *bpf_dup_subprog_starts(struct bpf_verifier_env *env) 977 + { 978 + u32 *starts = NULL; 979 + 980 + starts = kvmalloc_objs(u32, env->subprog_cnt, GFP_KERNEL_ACCOUNT); 981 + if (starts) { 982 + for (int i = 0; i < env->subprog_cnt; i++) 983 + starts[i] = env->subprog_info[i].start; 984 + } 985 + return starts; 986 + } 987 + 988 + static void bpf_restore_subprog_starts(struct bpf_verifier_env *env, u32 *orig_starts) 989 + { 990 + for (int i = 0; i < env->subprog_cnt; i++) 991 + env->subprog_info[i].start = orig_starts[i]; 992 + /* restore the start of fake 'exit' subprog as well */ 993 + env->subprog_info[env->subprog_cnt].start = env->prog->len; 994 + } 995 + 996 + static struct bpf_insn_aux_data *bpf_dup_insn_aux_data(struct bpf_verifier_env *env) 997 + { 998 + size_t size; 999 + void *new_aux; 1000 + 1001 + size = array_size(sizeof(struct bpf_insn_aux_data), env->prog->len); 1002 + new_aux = __vmalloc(size, GFP_KERNEL_ACCOUNT); 1003 + if (new_aux) 1004 + memcpy(new_aux, env->insn_aux_data, size); 1005 + return new_aux; 1006 + } 1007 + 1008 + static void bpf_restore_insn_aux_data(struct bpf_verifier_env *env, 1009 + struct bpf_insn_aux_data *orig_insn_aux) 1010 + { 1011 + /* the expanded elements are zero-filled, so no special handling is required */ 1012 + vfree(env->insn_aux_data); 1013 + env->insn_aux_data = orig_insn_aux; 1014 + } 1015 + 1016 + static int jit_subprogs(struct bpf_verifier_env *env) 977 1017 { 978 1018 struct bpf_prog *prog = env->prog, **func, *tmp; 979 1019 int i, j, subprog_start, subprog_end = 0, len, subprog; ··· 1021 981 struct bpf_insn *insn; 1022 982 void *old_bpf_func; 1023 983 int err, num_exentries; 1024 - int old_len, subprog_start_adjustment = 0; 1025 - 1026 - if (env->subprog_cnt <= 1) 1027 - return 0; 1028 984 1029 985 for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) { 1030 986 if (!bpf_pseudo_func(insn) && !bpf_pseudo_call(insn)) ··· 1089 1053 goto out_free; 1090 1054 func[i]->is_func = 1; 1091 1055 func[i]->sleepable = prog->sleepable; 1056 + func[i]->blinded = prog->blinded; 1092 1057 func[i]->aux->func_idx = i; 1093 1058 /* Below members will be freed only at prog->aux */ 1094 1059 func[i]->aux->btf = prog->aux->btf; 1095 - func[i]->aux->subprog_start = subprog_start + subprog_start_adjustment; 1060 + func[i]->aux->subprog_start = subprog_start; 1096 1061 func[i]->aux->func_info = prog->aux->func_info; 1097 1062 func[i]->aux->func_info_cnt = prog->aux->func_info_cnt; 1098 1063 func[i]->aux->poke_tab = prog->aux->poke_tab; ··· 1150 1113 func[i]->aux->token = prog->aux->token; 1151 1114 if (!i) 1152 1115 func[i]->aux->exception_boundary = env->seen_exception; 1153 - 1154 - /* 1155 - * To properly pass the absolute subprog start to jit 1156 - * all instruction adjustments should be accumulated 1157 - */ 1158 - old_len = func[i]->len; 1159 1116 func[i] = bpf_int_jit_compile(func[i]); 1160 - subprog_start_adjustment += func[i]->len - old_len; 1161 - 1162 1117 if (!func[i]->jited) { 1163 1118 err = -ENOTSUPP; 1164 1119 goto out_free; ··· 1276 1247 } 1277 1248 kfree(func); 1278 1249 out_undo_insn: 1250 + bpf_prog_jit_attempt_done(prog); 1251 + return err; 1252 + } 1253 + 1254 + int bpf_jit_subprogs(struct bpf_verifier_env *env) 1255 + { 1256 + int err, i; 1257 + bool blinded = false; 1258 + struct bpf_insn *insn; 1259 + struct bpf_prog *prog, *orig_prog; 1260 + struct bpf_insn_aux_data *orig_insn_aux; 1261 + u32 *orig_subprog_starts; 1262 + 1263 + if (env->subprog_cnt <= 1) 1264 + return 0; 1265 + 1266 + prog = orig_prog = env->prog; 1267 + if (bpf_prog_need_blind(prog)) { 1268 + orig_insn_aux = bpf_dup_insn_aux_data(env); 1269 + if (!orig_insn_aux) { 1270 + err = -ENOMEM; 1271 + goto out_cleanup; 1272 + } 1273 + orig_subprog_starts = bpf_dup_subprog_starts(env); 1274 + if (!orig_subprog_starts) { 1275 + vfree(orig_insn_aux); 1276 + err = -ENOMEM; 1277 + goto out_cleanup; 1278 + } 1279 + prog = bpf_jit_blind_constants(env, prog); 1280 + if (IS_ERR(prog)) { 1281 + err = -ENOMEM; 1282 + prog = orig_prog; 1283 + goto out_restore; 1284 + } 1285 + blinded = true; 1286 + } 1287 + 1288 + err = jit_subprogs(env); 1289 + if (err) 1290 + goto out_jit_err; 1291 + 1292 + if (blinded) { 1293 + bpf_jit_prog_release_other(prog, orig_prog); 1294 + kvfree(orig_subprog_starts); 1295 + vfree(orig_insn_aux); 1296 + } 1297 + 1298 + return 0; 1299 + 1300 + out_jit_err: 1301 + if (blinded) { 1302 + bpf_jit_prog_release_other(orig_prog, prog); 1303 + /* roll back to the clean original prog */ 1304 + prog = env->prog = orig_prog; 1305 + goto out_restore; 1306 + } else { 1307 + if (err != -EFAULT) { 1308 + /* 1309 + * We will fall back to interpreter mode when err is not -EFAULT, before 1310 + * that, insn->off and insn->imm should be restored to their original 1311 + * values since they were modified by jit_subprogs. 1312 + */ 1313 + for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) { 1314 + if (!bpf_pseudo_call(insn)) 1315 + continue; 1316 + insn->off = 0; 1317 + insn->imm = env->insn_aux_data[i].call_imm; 1318 + } 1319 + } 1320 + goto out_cleanup; 1321 + } 1322 + 1323 + out_restore: 1324 + bpf_restore_subprog_starts(env, orig_subprog_starts); 1325 + bpf_restore_insn_aux_data(env, orig_insn_aux); 1326 + kvfree(orig_subprog_starts); 1327 + out_cleanup: 1279 1328 /* cleanup main prog to be interpreted */ 1280 1329 prog->jit_requested = 0; 1281 1330 prog->blinding_requested = 0; 1282 - for (i = 0, insn = prog->insnsi; i < prog->len; i++, insn++) { 1283 - if (!bpf_pseudo_call(insn)) 1284 - continue; 1285 - insn->off = 0; 1286 - insn->imm = env->insn_aux_data[i].call_imm; 1287 - } 1288 - bpf_prog_jit_attempt_done(prog); 1289 1331 return err; 1290 1332 } 1291 1333