Memory limit causing hang?

[ackerleytng]

I was hoping to try https://www.fastly.com/blog/how-fuzz-server-american-fuzzy-lop without having to modify the code.

With the following memory setting (512mb),

/home/user/afl/afl-fuzz -m 512 -i /home/user/testcases -o /home/user/outputs -D 1 -t 200+ -N udp://localhost:53 /home/user/knot-2.1.1/src/knotd -c /home/user/knot.conf

But on inspection with strace, it seems like there isn't enough memory

3398  mmap(NULL, 524288000, PROT_READ, MAP_SHARED, 5, 0) = -1 ENOMEM (Cannot allocate memory)
...
3398  write(2, "2016-04-14T08:35:43 critical: failed to open configuration database '' (not enough memor
y)\n", 91) = 91
...
3398  exit_group(1)                     = ?
3398  +++ exited with 1 +++

Running it with more memory (1024mb) results in

$ /home/user/afl/afl-fuzz -m 1024 -i /home/user/testcases -o /home/user/outputs -D 1 -t 200+ -N udp://localhost:53 /home/user/knot-2.1.1/src/knotd -c /home/user/knot.conf
afl-fuzz 1.95b by <lcamtuf@google.com>
[+] You have 1 CPU cores and 2 runnable tasks (utilization: 200%).
[*] Checking core_pattern...
[*] Setting up output directories...
[+] Output directory exists but deemed OK to reuse.
[*] Deleting old session data...
[+] Output dir cleanup successful.
[*] Scanning '/home/user/testcases'...
[+] No auto-generated dictionary tokens to reuse.
[*] Creating hard links for all input files...
[*] Validating target binary...
[*] Attempting dry run with 'id:000000,orig:dns-query-sample'...
[*] Spinning up the fork server...
[+] All right - fork server is up.
[!] WARNING: Test case results in a hang (skipping)

[-] PROGRAM ABORT : All test cases time out, giving up!
         Location : perform_dry_run(), afl-fuzz.c:3240

Any ideas?

[jdbirdwell]

The first case does appear to be a problem with insufficient memory. The
second is occurring because the target is not exiting. Did you modify the
target to exit after processing the first received packet? Most daemons are
essentially infinite loops, waiting forever to process requests as they
arrive and must be modified to process only one request and then exit.

Doug Birdwell

On Wed, Apr 13, 2016, 8:44 PM Ackerley Tng notifications@github.com wrote:

I was hoping to try
https://www.fastly.com/blog/how-fuzz-server-american-fuzzy-lop without
having to modify the code.

With the following memory setting (512mb),

/home/user/afl/afl-fuzz -m 512 -i /home/user/testcases -o /home/user/outputs -D 1 -t 200+ -N udp://localhost:53 /home/user/knot-2.1.1/src/knotd -c /home/user/knot.conf

But on inspection with strace, it seems like there isn't enough memory

3398 mmap(NULL, 524288000, PROT_READ, MAP_SHARED, 5, 0) = -1 ENOMEM (Cannot allocate memory)
...
3398 write(2, "2016-04-14T08:35:43 critical: failed to open configuration database '' (not enough memor
y)\n", 91) = 91
...
3398 exit_group(1) = ?
3398 +++ exited with 1 +++

Running it with more memory (1024mb) results in

$ /home/user/afl/afl-fuzz -m 1024 -i /home/user/testcases -o /home/user/outputs -D 1 -t 200+ -N udp://localhost:53 /home/user/knot-2.1.1/src/knotd -c /home/user/knot.conf
afl-fuzz 1.95b by lcamtuf@google.com
[+] You have 1 CPU cores and 2 runnable tasks (utilization: 200%).
[] Checking core_pattern...
[] Setting up output directories...
[+] Output directory exists but deemed OK to reuse.
[] Deleting old session data...
[+] Output dir cleanup successful.
[] Scanning '/home/user/testcases'...
[+] No auto-generated dictionary tokens to reuse.
[] Creating hard links for all input files...
[] Validating target binary...
[] Attempting dry run with 'id:000000,orig:dns-query-sample'...
[] Spinning up the fork server...
[+] All right - fork server is up.
[!] WARNING: Test case results in a hang (skipping)

[-] PROGRAM ABORT : All test cases time out, giving up!
Location : perform_dry_run(), afl-fuzz.c:3240

Any ideas?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
#2

[ackerleytng]

I see, let me try that out and i'll let you know. Thanks!

[ackerleytng]

After inserting code to force an exit, I'm still seeing the same hang.

I pushed the memory limit up to 2048.

# strace -f -s100 -o out.txt /home/user/afl/afl-fuzz -m 2048 -i /home/user/testcases -o /home/user/outputs -D 1 -t 200+ -N udp://localhost:53 /home/user/knot-2.1.1/src/knotd -c /home/user/knot.conf
afl-fuzz 1.95b by <lcamtuf@google.com>
[+] You have 1 CPU cores and 2 runnable tasks (utilization: 200%).
[*] Checking core_pattern...
[*] Setting up output directories...
[+] Output directory exists but deemed OK to reuse.
[*] Deleting old session data...
[+] Output dir cleanup successful.
[*] Scanning '/home/user/testcases'...
[+] No auto-generated dictionary tokens to reuse.
[*] Creating hard links for all input files...
[*] Validating target binary...
[*] Attempting dry run with 'id:000000,orig:dns-query-sample'...
[*] Spinning up the fork server...
[+] All right - fork server is up.
[!] WARNING: Test case results in a hang (skipping)

[-] PROGRAM ABORT : All test cases time out, giving up!
         Location : perform_dry_run(), afl-fuzz.c:3240

root@afl-network-knotdns:/home/user# grep mmap out.txt | grep ENOMEM
root@afl-network-knotdns:/home/user#

I got this in out.txt

47326 futex(0x11cbf64, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
47320 <... read resumed> 0x7ffc6a02ec60, 4) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
47320 --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
47320 kill(47325, SIGKILL)              = 0
47340 +++ killed by SIGKILL +++
47339 +++ killed by SIGKILL +++
47338 +++ killed by SIGKILL +++
47337 +++ killed by SIGKILL +++
47336 +++ killed by SIGKILL +++
47335 +++ killed by SIGKILL +++
47334 +++ killed by SIGKILL +++
47333 +++ killed by SIGKILL +++
47332 +++ killed by SIGKILL +++
47331 +++ killed by SIGKILL +++
47330 +++ killed by SIGKILL +++
47328 +++ killed by SIGKILL +++
47327 +++ killed by SIGKILL +++
47326 +++ killed by SIGKILL +++
47320 rt_sigreturn()                    = 0
47320 read(8,  <unfinished ...>
47329 +++ killed by SIGKILL +++
47325 +++ killed by SIGKILL +++

[jdbirdwell]

I believe this is the shared memory area that afl uses to allow targets to
record branch data. Are you running a 64 bit process? Someone else may want
to chime in, but i vaguely remember there are cases (or used to be) where
afl required an excessive memory mapped region on 64 bit machines. I don't
think this is caused by the networking code. What happens when you give it
even more memory?

Doug

On Wed, Apr 13, 2016, 10:09 PM Ackerley Tng notifications@github.com
wrote:

After inserting code to force an exit, I'm still seeing the same hang.

I looked into strace and spotted lots of

33630 mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0 <unfinished ...>
...
33630 mmap(NULL, 134217728, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Cann
ot allocate memory)
33630 mmap(NULL, 67108864, PROT_NONE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = -1 ENOMEM (Canno
t allocate memory)
33630 sendto(8, "out of memory [33617]", 21, MSG_NOSIGNAL, NULL, 0) = 21
...
33618 rt_sigprocmask(SIG_UNBLOCK, [ALRM], NULL, 8) = 0
33618 futex(0x1ee3f64, FUTEX_WAIT_PRIVATE, 2, NULL <unfinished ...>
33615 <... read resumed> 0x7ffeaffe57c0, 4) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
33615 --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
33615 kill(33617, SIGKILL) = 0
33632 +++ killed by SIGKILL +++
33631 +++ killed by SIGKILL +++
33630 +++ killed by SIGKILL +++
33629 +++ killed by SIGKILL +++
33628 +++ killed by SIGKILL +++
33627 +++ killed by SIGKILL +++
33626 +++ killed by SIGKILL +++
33625 +++ killed by SIGKILL +++
33624 +++ killed by SIGKILL +++
33623 +++ killed by SIGKILL +++
33622 +++ killed by SIGKILL +++
33620 +++ killed by SIGKILL +++
33619 +++ killed by SIGKILL +++
33618 +++ killed by SIGKILL +++
33615 rt_sigreturn() = 0
33615 read(8, <unfinished ...>
33621 +++ killed by SIGKILL +++
33617 +++ killed by SIGKILL +++

I tried to make it print before exiting but I think it didn't reach that
part of the code.

—
You are receiving this because you commented.

Reply to this email directly or view it on GitHub
#2 (comment)

[ackerleytng]

Thanks for your quick replies!!

I'm at 65536mb, I don't think memory is the issue anymore.

root@afl-network-knotdns:/home/user# /home/user/afl/afl-fuzz -m 65536 -i /home/user/testcases -o /home/user/outputs -D 1 -t 200+ -N udp://localhost:53 /home/user/knot-2.1.1/src/knotd -c /home/user/knot.conf
afl-fuzz 1.95b by <lcamtuf@google.com>
[+] You have 1 CPU cores and 2 runnable tasks (utilization: 200%).
[*] Checking core_pattern...
[*] Setting up output directories...
[+] Output directory exists but deemed OK to reuse.
[*] Deleting old session data...
[+] Output dir cleanup successful.
[*] Scanning '/home/user/testcases'...
[+] No auto-generated dictionary tokens to reuse.
[*] Creating hard links for all input files...
[*] Validating target binary...
[*] Attempting dry run with 'id:000000,orig:dns-query-sample'...
[*] Spinning up the fork server...
[+] All right - fork server is up.
[!] WARNING: Test case results in a hang (skipping)

[-] PROGRAM ABORT : All test cases time out, giving up!
         Location : perform_dry_run(), afl-fuzz.c:3240

This time I grepped only the pid that returned ERESTARTSYS

47320 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 200000}}, NULL) = 0
47320 read(8,  <unfinished ...>
47320 <... read resumed> 0x7ffc6a02ec60, 4) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
47320 --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
47320 kill(47325, SIGKILL)              = 0
47320 rt_sigreturn()                    = 0
47320 read(8,  <unfinished ...>
47320 <... read resumed> "\t\0\0\0", 4) = 4
47320 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 0}},  <unfinished ...>
47320 <... setitimer resumed> NULL)     = 0
47320 write(1, "\33[1;33m[!] \33[1;37mWARNING: \33[0mTest case results in a hang (skipping)\33[0m\n", 74) = 74

[jdbirdwell]

I suspect you are correct, but it's late here. Tomorrow...

Doug

On Wed, Apr 13, 2016, 10:22 PM Ackerley Tng notifications@github.com
wrote:

Thanks for your quick replies!!

I'm at 65536mb, I don't think memory is the issue anymore.

root@afl-network-knotdns:/home/user# /home/user/afl/afl-fuzz -m 65536 -i /home/user/testcases -o /home/user/outputs -D 1 -t 200+ -N udp://localhost:53 /home/user/knot-2.1.1/src/knotd -c /home/user/knot.conf
afl-fuzz 1.95b by lcamtuf@google.com
[+] You have 1 CPU cores and 2 runnable tasks (utilization: 200%).
[] Checking core_pattern...
[] Setting up output directories...
[+] Output directory exists but deemed OK to reuse.
[] Deleting old session data...
[+] Output dir cleanup successful.
[] Scanning '/home/user/testcases'...
[+] No auto-generated dictionary tokens to reuse.
[] Creating hard links for all input files...
[] Validating target binary...
[] Attempting dry run with 'id:000000,orig:dns-query-sample'...
[] Spinning up the fork server...
[+] All right - fork server is up.
[!] WARNING: Test case results in a hang (skipping)

[-] PROGRAM ABORT : All test cases time out, giving up!
Location : perform_dry_run(), afl-fuzz.c:3240

This time I grepped only the pid that returned ERESTARTSYS

47320 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 200000}}, NULL) = 0
47320 read(8, <unfinished ...>
47320 <... read resumed> 0x7ffc6a02ec60, 4) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
47320 --- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---
47320 kill(47325, SIGKILL) = 0
47320 rt_sigreturn() = 0
47320 read(8, <unfinished ...>
47320 <... read resumed> "\t\0\0\0", 4) = 4
47320 setitimer(ITIMER_REAL, {it_interval={0, 0}, it_value={0, 0}}, <unfinished ...>
47320 <... setitimer resumed> NULL) = 0
47320 write(1, "\33[1;33m[!] \33[1;37mWARNING: \33[0mTest case results in a hang (skipping)\33[0m\n", 74) = 74

—
You are receiving this because you commented.

Reply to this email directly or view it on GitHub
#2 (comment)