Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

invalid mem access when navigating struct sock #253

Closed
brendangregg opened this issue Sep 25, 2015 · 5 comments · Fixed by #1285
Closed

invalid mem access when navigating struct sock #253

brendangregg opened this issue Sep 25, 2015 · 5 comments · Fixed by #1285
Labels

Comments

@brendangregg
Copy link
Member

Here is a program that I thought should work:

#!/usr/bin/python
from bcc import BPF
bpf_text = """
#include <uapi/linux/ptrace.h>
#include <net/sock.h>
#include <bcc/proto.h>

BPF_HASH(currsock, u32, struct sock *);

int kprobe__tcp_v4_connect(struct pt_regs *ctx, struct sock *sk,
    struct sockaddr *uaddr, int addr_len)
{
    u32 pid = bpf_get_current_pid_tgid();
    currsock.update(&pid, &sk);
    return 0;
};

int kretprobe__tcp_v4_connect(struct pt_regs *ctx)
{
    u32 pid = bpf_get_current_pid_tgid();
    struct sock **skpp;
    skpp = currsock.lookup(&pid);
    if (skpp) {
        struct sock *skp = *skpp;
        bpf_trace_printk("addr: %lx\\n", skp->__sk_common.skc_daddr);
        currsock.delete(&pid);
    }
    return 0;
}
"""
b = BPF(text=bpf_text)
print("running"); b.trace_print(); exit()

But won't run:

# ./tcpv4connect
bpf: Permission denied
0: (85) call 14
1: (63) *(u32 *)(r10 -4) = r0
2: (18) r1 = 0xe2f31180
4: (bf) r2 = r10
5: (07) r2 += -4
6: (85) call 1
7: (15) if r0 == 0x0 goto pc+18
 R0=map_value(ks=4,vs=8) R10=fp
8: (79) r1 = *(u64 *)(r0 +0)
9: (b7) r2 = 0
10: (73) *(u8 *)(r10 -12) = r2
11: (b7) r2 = 175664165
12: (63) *(u32 *)(r10 -16) = r2
13: (18) r2 = 0x75746572
15: (7b) *(u64 *)(r10 -24) = r2
16: (61) r3 = *(u32 *)(r1 +0)
R1 invalid mem access 'inv'

Traceback (most recent call last):
  File "./tcpv4connect.2", line 30, in <module>
    b = BPF(text=bpf_text)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 383, in __init__
    self._trace_autoload()
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 610, in _trace_autoload
    fn = self.load_func(func_name, BPF.KPROBE)
  File "/usr/lib/python2.7/dist-packages/bcc/__init__.py", line 418, in load_func
    raise Exception("Failed to load BPF program %s" % func_name)
Exception: Failed to load BPF program kretprobe__tcp_v4_connect

It works if I use this instead, in the kretprobe:

[...]
        if (skpp) {
                struct sock *skp = *skpp;
                u32 daddr = 0;
                bpf_probe_read(&daddr, sizeof(daddr), &skp->__sk_common.skc_daddr);
                bpf_trace_printk("addr: %x\\n", daddr);
[...]

Is needing an explicit bpf_probe_read() a bug or is it supposed to be this way? If it is supposed to be this way, then we'll need to document the logic to make it easier to follow...

@yonghong-song
Copy link
Collaborator

From bpf verifier point of view, for the code:

int kretprobe__tcp_v4_connect(struct pt_regs _ctx)
{
u32 pid = bpf_get_current_pid_tgid();
struct sock *_skpp;
skpp = currsock.lookup(&pid);
if (skpp) {
struct sock *skp = *skpp;
bpf_trace_printk("addr: %lx\n", skp->__sk_common.skc_daddr);
currsock.delete(&pid);
}
return 0;
}

After "skp = *skpp", skp becomes an unknown value since the verifier has no idea of
what the *skpp could be. hence later skp->__sk_common.skc_daddr will have
verification error.

I guess in this case that bcc rewriter could automatically convert the skp->__sk_common.sdk_addr to a bpf_probe_read since it can recognize the map value is dereferenced.

@drzaeus77
Copy link
Collaborator

Yes, I think I know what's missing in the rewriter. Shouldn't be too hard.
On Sep 26, 2015 1:51 AM, "yonghong-song" notifications@github.com wrote:

From bpf verifier point of view, for the code:

int kretprobe__tcp_v4_connect(struct pt_regs

_ctx) { u32 pid = bpf_get_current_pid_tgid(); struct sock *_skpp;
skpp = currsock.lookup(&pid);
if (skpp) {
struct sock *skp = *skpp;
bpf_trace_printk("addr: %lx\n", skp->__sk_common.skc_daddr);
currsock.delete(&pid);
}
return 0;
}

After "skp = *skpp", skp becomes an unknown value since the verifier has
no idea of
what the *skpp could be. hence later skp->__sk_common.skc_daddr will have
verification error.

I guess in this case that bcc rewriter could automatically convert the
skp->__sk_common.sdk_addr to a bpf_probe_read since it can recognize the
map value is dereferenced.


Reply to this email directly or view it on GitHub
#253 (comment).

@brendangregg
Copy link
Member Author

Thanks; navigating from sk to skc_daddr will likely be common, so would be good to improve.

PS, for anyone trying to understand this code... I'm doing it on kretprobe instead of kprobe for two reasons:

  • I wanted the return status of the TCP connection (eg, ECONNREFUSED). However, ctx->ax in kretprobe isn't what I hoped it would be: it is always 0.
  • Fetching skc_daddr on tcp_v4_connect on kprobe was always zero. I assume it's not populated on entry, but haven't read all the code to confirm. The sk is certainly populated on return.

@yonghong-song
Copy link
Collaborator

@brendangregg , for your question above ctx->ax is always 0.
tcp_v4_connect function returning 0 means the SYNC has been successfully sent to the other side.
If SYNC is not able to sent out to the other side, ctx->ax will not be 0.

On the other hand, ctx->ax=0 in tcp_v4_connect does not mean the syscall "connect" will succeed.
If the remote site sends back a RST, the connect will fail.

In your code, I guess you still want to check ctx->ax. Otherwise, the value you tried to print may not
be populated.

@mikedanese
Copy link

mikedanese commented Aug 3, 2016

I'm trying to implement

        if (ip_hdr(skb)->protocol!=IPPROTO_TCP)
                return 0;

and getting a similar error:

0: (79) r1 = *(u64 *)(r1 +104)
1: (69) r2 = *(u16 *)(r1 +196)
R1 invalid mem access 'inv'

This operation seems simple enough but after two inlined function, it expands to a number of derefences.

        struct iphdr *iph = (struct iphdr *)(skb->head + skb->network_header);
        if (iph->protocol!=IPPROTO_TCP)
                return 0;

Which fails with a similar error.

The doc for bpf_probe_read says

This happens automatically in some cases, such as dereferencing kernel variables, as bcc will rewrite the BPF program to include the necessary bpf_probe_reads().

Why is this case not covered by the compiler? Do I need to use bpf_probe_read for every dereference?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants