Raspbian: swapper page allocation failure #153
Comments
|
These Pandaboard/Beagleboard bug reports may be relevant: Strange out of memory on pandaboard The culprit seems to be the usbnet driver, specifically turbo mode. Unfortunately the proposed solutions are to cut USB/eth performance by disabling turbo mode, or bump up the minimum free memory (I've already tried 32MB, no luck). Does the network-over-USB really need this much memory to function properly? |
|
I now see this has already been discussed previously, issue #9 Is the patch from Ming Lei applicable to the Pi? |
|
The Ming Lei patch looks plausible. Have you tried it? I just tried applying it, and it applies and builds. Can you test it? |
|
Last time I tried the Ming Lei patch it didn't solve the issue for me, but that was a very long time ago and it's very possible there were other factors involved. I'm also very interested in whether Dom's updated kernel.img is better for you. |
|
Just downloaded it, will give it a go and report back. Many thanks. |
|
Argh... I haven't even got around to testing the torrent download yet and I've already got a mass of "kevent 2" messages being spammed to syslog/kern.log - it's not looking good. Between 18:21:17 and 18:25:20 this patched kernel spammed 606,002 I need to stop this test before it kills my SD card... sorry, reverting to release kernel.img. |
|
Can you try: I'm not concinved there is any difference in performance, and it will probably fix this problem. See: |
|
Do you want me to try that with the patched kernel, or stock kernel? |
|
The stock kernel. The patched kernel doesn't work... |
|
I'd say iperf is a bit slower: smsc95xx.turbo_mode=Y smsc95xx.turbo_mode=N Testing Transmission will take a little while... fingers crossed... |
|
So far so good with smsc95xx.turbo_mode=N. I have 5 (FIVE!) concurrent torrents maxing out my 20Mbit/s connection, and after 15 minutes I have no allocation failures (or kevent 2 spam). Will allow the torrents to complete, check there are no errors and post an update. However, I was under the impression that disabling turbo_mode was bad for performance, but apparently not (or at least, not significantly). Maybe disabling it should be the default until such time as turbo_mode can be made reliable? |
|
Yes, I asked the question if turbo_mode=N caused a problem for anyone in the linked thread, and there didn't seem to be any problems. It seems to have a fairly minor performance hit (I thought my iperf drop was less than MillhouseVH's.) I think it's worth considering as a default. |
|
I'm afraid I missed that thread, but I'll post to it once I've completed this test. |
|
Yes, I had been under the impression that issues with turbo_mode no longer came up in normal usage. However, if issues like this are trivially reproducible with a torrent client it probably makes a lot of sense to disable turbo_mode by default. If anyone has use cases where they suffer a significant degradation with turbo_mode disabled do let us know. |
|
@MillHouseVH |
|
My XBMC of choice is OpenELEC, not sure if that would support this option - its cmdline.txt looks to be non-standard. |
|
I would expect it to work. Just add |
|
Is there any way to tell if the option has been honoured, as I don't normally see kevent2/swapper problems in OpenELEC so not seeing a problem won't mean much... |
|
Try copying a large file over network, the speed should be reduced a little. |
|
This is with OpenELEC build r12322, over NFS: #smsc95xx.turbo_mode=N #smsc95xx.turbo_mode=Y About the same ~1.5MB/s difference from the iperf test. So it looks like OpenELEC is honouring the option, and I couldn't see any obvious difference when playing back a 1080p/DTS5.1 movie - it played without any stutter or buffering. However if the bitrate where high enough, maybe it would be the difference between stutter free playback and not (OE should probably stick to keeping turbo_mode enabled by default, IMHO). Obviously the ideal scenario would be to have turbo_mode enabled, but without these allocation failures... I'm assuming this isn't possible, or likely? |
|
All 5 torrents completed without any error - looks like smsc95xx.turbo_mode=N is a keeper! |
|
I'm the author of the smsc95xx driver, and I would recommend turning this feature off on a slower Host CPU like Raspberry Pi's. This feature enables the USB device to send multiple received ethernet packets in one USB transaction. This enables more efficient usage of the USB bus, and therefore higher throughput, but it causes multiple RAM allocation requests to happen at once as multiple incoming packets are processed. Turning it off means memory allocations happen in a less bursty fashion, so are less likely to fail when RAM is scarce. I hope this helps. |
|
@MilhouseVH and others: Can you try transmission with vm.min_free_kbytes reduced to 2048 or 1024? |
|
Test setup:
smsc95xx.turbo_mode=N, vm.min_free_kbytes=1024 Started well, but after about 20 minutes a series of page allocation failures occurred, each followed by multiple "kevent 2 may have been dropped" messages. Result: FAIL smsc95xx.turbo_mode=N, vm.min_free_kbytes=2048 All downloads completed successfully after 45 minutes, and with no errors. Result: SUCCESS So, 2048 looks good. To see if 2048 was at all marginal, I ran the same Transmission test, but this time with 10 concurrent downloads (again, each ~700MB, average 25 peers per torrent) and this also completed without any errors. Finally, I tried the same 10 concurrent downloads, this time with |
|
Thanks. That's useful to know. |
|
Closing, as smsc95xx.turbo_mode=N seems to be the correct workaround. |
|
Overnight I had 16 torrents downloading continuously for several hours, with vm.min_free_kbytes=1536 and smsc95xx.turbo_mode=N, and experienced no problems whatsoever so unless you're being cautious and fancy some extra headroom, 1536 seems like an OK figure. |
|
Thanks for info. |
|
Testing with the latest kernel & firmware: With
I'm not really sure that 10240 is "safe", it's just a figure that in my very limited testing was sufficient at this point in time. |
|
This issue is still outstanding. With vm.min_free_kbytes=16536 and smsc95xx.turbo_mode=N I'm still seeing allocation failures on a Raspberry Pi Model B Revision 2. This is occuring with nothing more than lightweight network traffic - I'm running wget to download some PDF files at a peak rate of around 300Kb/s. I'm only running a single wget process at a time, so there's only ever one HTTP connection - meaning it's not a case of flooding the Pi with BitTorrent traffic, but a flaw that can be exploited even in a light usage scenario. Kernel version 3.6.11+ #399. |
|
@ghalfacree Can you produce a set of instructions for reproducing this? E.g. start with fresh raspbian image, and produce a sequence of commands that results in this failure? |
|
@popcornmix It's not easy, sadly: the problem only tends to crop up after the device has been running for quite some time - the first hit in the kernel log, pasted above, is from running the device solidly for nearly five days. I've upgraded to the very latest firmware and kernel - 3.6.11+ #403 - and will monitor it. For what it's worth, my usage is as follows: CPU usage often hits 100% during SSH backup, but I use arcfour when doing big transfers and it sits at around 70% then. I guess, compared to other's usage of the Pi, I could be stressing the USB bus more than most: all network data is either being read from or written to the two hard drives simultaneously. I've boosted vm.min_free_kbytes by 10MB, and I'll keep monitoring the situation. If it happens again - and I should know in a week, from past experience - I'll set up a fresh install with a single USB hard drive and see if I can reliably trigger the problem on that, too. |
|
Also note that the callstack is different to before, this call stack includes the new tasklet from P33M, it is possible there is some sort of locking problem between the tasklet and the main code… Gordon On 5 Apr 2013, at 14:21, Gareth Halfacree <[email protected]mailto:[email protected]> @popcornmixhttps://github.com/popcornmix It's not easy, sadly: the problem only tends to crop up after the device has been running for quite some time - the first hit in the kernel log, pasted above, is from running the device solidly for nearly five days. I've upgraded to the very latest firmware and kernel - 3.6.11+ #403 - and will monitor it. For what it's worth, my usage is as follows: CPU usage often hits 100% during SSH backup, but I use arcfour when doing big transfers and it sits at around 70% then. I guess, compared to other's usage of the Pi, I could be stressing the USB bus more than most: all network data is either being read from or written to the two hard drives simultaneously. I've boosted vm.min_free_kbytes by 10MB, and I'll keep monitoring the situation. If it happens again - and I should know in a week, from past experience - I'll set up a fresh install with a single USB hard drive and see if I can reliably trigger the problem on that, too. — |
|
@ghalfacree (the last one before the USB tasklet was introcuded) is it any better? |
|
I've been able to get it to fail after just one day, during sustained wget combined with heavy SSH uploading (it's about a third the way through a 15GB upload.) Once the transfer has completed, I'll revert to the kernel suggested by @popcornmix and try again. |
|
Using the kernel from commit aea9d8e529baf2c11e0282706b71ae8cfc8ee909, the system dropped off the network entirely this morning. As I'm running the unit headless, I was unable to view the kernel messages for a reason. I'm upgrading back to the latest kernel version, as I'd rather have recoverable errors than the entire thing become completely unresponsive to network traffic. EDIT: Key log extracts below: |
|
Back on the latest kernel, I can reliably trigger the allocation failures by just shoving a fair amount of sustained traffic through the Pi. That is followed by repeated page allocation failures - up to 100 at the time of posting. The affected process varies: it starts with sshd (as I'm transferring files via ssh), then rsyslogd, then ksoftirqd/0, kswapd0, usb-storage, swapper. Strangely, at no point is the system actually running out of memory - so why it's even trying to swap I don't know. Linux keylime 3.6.11+ #403 PREEMPT Tue Apr 2 22:48:13 BST 2013 armv6l GNU/Linux. Command line is: |
|
Have you tried with the commandline option 'elevator=noop'? elevator=dealine and elevator=cfq by default try to optimize block I/O for disk drives, not flash memory. While I don't think that this is the cause of the problem, it does affect how the kernel uses the swap space if you are swapping off a flash drive. As for the swapping, vm.min_free_kbytes=16536 tells the kernel to try to keep at least 16M of memory free. For some reason, your page cache appears to eating a huge ammount of memory, if the kernel isn't trying to reclaim space from the cache, then it will be trying to swap instead. You might try adding: |
|
@Ferroin: Swappiness is already set to 1 by default in Raspbian, so setting it to 30 would increase, rather than decrease, how much it tries to swap out of memory. For now, I've tried turning swap off entirely in order to see what effect that has on the system. I should know in the next couple of days whether it has made any difference. Should it show the same symptoms even without a swap device, I'll try playing with the kernel options you've suggested. |
|
@ghalfacree: I did not know that, as I use Gentoo, not Raspbian. Have you tried using ext4 instead of btrfs? btrfs is unstable after all, and it has a tendency to put pretty heavy load on lower bandwidth busses like USB. |
|
@Ferroin: Ext4 does not support the features I require - the discussion of which is outside the scope of this issue. Btrfs may be unstable - although I think its developers would prefer you use the term "experimental" - but the fact is it's not btrfs that's failing here: it's the Raspbian kernel. I have used btrfs 0.19 on similar hardware in the past with no issues whatsoever. |
|
@gordonhollingworth The source of the allocation failure (top of the call stack) is still constant after the tasklet patch. The reason you're seeing the tasklet appear is because it is now the calling function for usb_hcd_giveback_urb rather than the interrupt handler directly. The tasklet is interrupt-safe in data transfer between the IRQ and tasklet - I have seen instances where it gets pre-empted as much as 10 times in a single run during boot. I did try to find the source of page allocation errors in the process of finding the cause of #189 but the frustratingly infrequent occurrences and disappearance when I turned on SLUB object owner tracking made me give up. If btrfs gives a more repeatable result then I could probably start digging again. On the latest kernel during a bit of stress testing I did try a maxed out torrent at the speed of my line. 2MB/s to USB HDD, no issues during the 30minute transfer. |
|
Since turning swap off entirely, I have not had any page allocation failures - unsurprisingly. @P33M: It certainly seems that running a pair of external harddrives set up in a RAID1 array through btrfs and shoving a bunch of data at them is enough to reliably trigger the bug within a single day. To match my setup, mount the btrfs drives with LZO compression and free space caching enabled, and transfer the data via SSH. You'll certainly need to spend more than half an hour running the test, though: the earliest I've been able to get it to trigger is after seven hours. |
|
Similar errors with 3.2.x series kernel are intereseting? (yeah, I know, I should upgrade, but my pi has 54 days uptime and I just don't want to brake it :P ) |
|
Any progress on this? The errors occur even when swap is deactivated, as per the below log: |
popcornmix
pushed a commit
that referenced
this issue
Jan 22, 2014
…resume There are cases where cpufreq_add_dev() may fail for some CPUs during system resume. With the current code we will still have sysfs cpufreq files for those CPUs and struct cpufreq_policy would be already freed for them. Hence any operation on those sysfs files would result in kernel warnings. Example of problems resulting from resume errors (from Bjørn Mork): WARNING: CPU: 0 PID: 6055 at fs/sysfs/file.c:343 sysfs_open_file+0x77/0x212() missing sysfs attribute operations for kobject: (null) Modules linked in: [stripped as irrelevant] CPU: 0 PID: 6055 Comm: grep Tainted: G D 3.13.0-rc2 #153 Hardware name: LENOVO 2776LEG/2776LEG, BIOS 6EET55WW (3.15 ) 12/19/2011 0000000000000009 ffff8802327ebb78 ffffffff81380b0e 0000000000000006 ffff8802327ebbc8 ffff8802327ebbb8 ffffffff81038635 0000000000000000 ffffffff811823c7 ffff88021a19e688 ffff88021a19e688 ffff8802302f9310 Call Trace: [<ffffffff81380b0e>] dump_stack+0x55/0x76 [<ffffffff81038635>] warn_slowpath_common+0x7c/0x96 [<ffffffff811823c7>] ? sysfs_open_file+0x77/0x212 [<ffffffff810386e3>] warn_slowpath_fmt+0x41/0x43 [<ffffffff81182dec>] ? sysfs_get_active+0x6b/0x82 [<ffffffff81182382>] ? sysfs_open_file+0x32/0x212 [<ffffffff811823c7>] sysfs_open_file+0x77/0x212 [<ffffffff81182350>] ? sysfs_schedule_callback+0x1ac/0x1ac [<ffffffff81122562>] do_dentry_open+0x17c/0x257 [<ffffffff8112267e>] finish_open+0x41/0x4f [<ffffffff81130225>] do_last+0x80c/0x9ba [<ffffffff8112dbbd>] ? inode_permission+0x40/0x42 [<ffffffff81130606>] path_openat+0x233/0x4a1 [<ffffffff81130b7e>] do_filp_open+0x35/0x85 [<ffffffff8113b787>] ? __alloc_fd+0x172/0x184 [<ffffffff811232ea>] do_sys_open+0x6b/0xfa [<ffffffff811233a7>] SyS_openat+0xf/0x11 [<ffffffff8138c812>] system_call_fastpath+0x16/0x1b To fix this, remove those sysfs files or put the associated kobject in case of such errors. Also, to make it simple, remove the cpufreq sysfs links from all the CPUs (except for the policy->cpu) during suspend, as that operation won't result in a loss of sysfs file permissions and we can create those links during resume just fine. Fixes: 5302c3f ("cpufreq: Perform light-weight init/teardown during suspend/resume") Reported-and-tested-by: Bjørn Mork <[email protected]> Signed-off-by: Viresh Kumar <[email protected]> Cc: 3.12+ <[email protected]> # 3.12+ [rjw: Changelog] Signed-off-by: Rafael J. Wysocki <[email protected]>
|
Memory fragmentation is the root cause of this issue. CONFIG_COMPACTION and CONFIG_SLUB make this orders of magnitude less likely to happen. |
popcornmix
pushed a commit
that referenced
this issue
Jun 8, 2014
... at edp_have_panel_vdd. Just return false, saying we don't have the panel VDD since the device is suspended. We started getting WARNs about this problem since the patch that started checking if we're suspended while reading registers. Example backtrace provided by Paulo: [ 63.572201] [drm:hsw_enable_pc8] Enabling package C8+ [ 63.581831] [drm:i915_runtime_suspend] Device suspended [ 63.664798] ------------[ cut here ]------------ [ 63.664824] WARNING: CPU: 3 PID: 828 at drivers/gpu/drm/i915/intel_uncore.c:47 assert_device_not_suspended.isra.7+0x32/0x40 [i915]() [ 63.664826] Device suspended [ 63.664828] Modules linked in: ccm fuse ip6table_filter ip6_tables ebtable_nat ebtables arc4 ath9k_htc ath9k_common ath9k_hw mac80211 ath cfg80211 iTCO_wdt iTCO_vendor_support x86_pkg_temp_thermal coretemp microcode i2c_i801 e1000e pcspkr serio_raw lpc_ich ptp pps_core mei_me mei mfd_core dm_crypt i915 crc32_pclmul crc32c_intel ghash_clmulni_intel i2c_algo_bit drm_kms_helper drm video [ 63.664867] CPU: 3 PID: 828 Comm: kworker/3:3 Not tainted 3.14.0+ #153 [ 63.664869] Hardware name: Intel Corporation Shark Bay Client platform/WhiteTip Mountain 1, BIOS HSWLPTU1.86C.0133.R00.1309172123 09/17/2013 [ 63.664887] Workqueue: events edp_panel_vdd_work [i915] [ 63.664889] 0000000000000009 ffff88009d745c28 ffffffff8167ec6f ffff88009d745c70 [ 63.664895] ffff88009d745c60 ffffffff8106c8ed ffff880036278000 00000000000c7204 [ 63.664900] ffff88014f2d3040 ffff880036278070 0000000000000001 ffff88009d745cc0 [ 63.664905] Call Trace: [ 63.664911] [<ffffffff8167ec6f>] dump_stack+0x4d/0x66 [ 63.664916] [<ffffffff8106c8ed>] warn_slowpath_common+0x7d/0xa0 [ 63.664920] [<ffffffff8106c95c>] warn_slowpath_fmt+0x4c/0x50 [ 63.664926] [<ffffffff810bd6be>] ? mark_held_locks+0xae/0x130 [ 63.664941] [<ffffffffa00d80d2>] assert_device_not_suspended.isra.7+0x32/0x40 [i915] [ 63.664956] [<ffffffffa00d99d2>] gen6_read32+0x32/0x120 [i915] [ 63.664969] [<ffffffffa00d99a0>] ? gen6_read8+0x120/0x120 [i915] [ 63.664985] [<ffffffffa0106f8f>] edp_have_panel_vdd+0x3f/0x50 [i915] [ 63.665000] [<ffffffffa01074e8>] edp_panel_vdd_off_sync+0x58/0x1c0 [i915] [ 63.665004] [<ffffffff8108a06c>] ? process_one_work+0x18c/0x560 [ 63.665018] [<ffffffffa0107684>] edp_panel_vdd_work+0x34/0x50 [i915] [ 63.665022] [<ffffffff8108a0d7>] process_one_work+0x1f7/0x560 [ 63.665026] [<ffffffff8108a06c>] ? process_one_work+0x18c/0x560 [ 63.665031] [<ffffffff8108ae2b>] worker_thread+0x11b/0x3a0 [ 63.665035] [<ffffffff8108ad10>] ? manage_workers.isra.21+0x2a0/0x2a0 [ 63.665039] [<ffffffff810916fc>] kthread+0xfc/0x120 [ 63.665043] [<ffffffff81091600>] ? kthread_create_on_node+0x230/0x230 [ 63.665048] [<ffffffff8169082c>] ret_from_fork+0x7c/0xb0 [ 63.665052] [<ffffffff81091600>] ? kthread_create_on_node+0x230/0x230 [ 63.665054] ---[ end trace 1250bcc890af9999 ]--- [ 63.665060] [drm:edp_panel_vdd_off_sync] Turning eDP VDD off [ 63.665061] ------------[ cut here ]------------ Testcase: igt/pm_pc8 Signed-off-by: Paulo Zanoni <[email protected]> Signed-off-by: Daniel Vetter <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Oct 8, 2014
Fixes the following checkpatch warnings:
WARNING: please, no space before tabs
#5: FILE: drivers/mfd/88pm860x-core.c:5:
+ * ^IHaojian Zhuang <[email protected]>$
WARNING: line over 80 characters
#143: FILE: drivers/mfd/88pm860x-core.c:143:
+ {PM8607_IRQ_AUDIO_SHORT, PM8607_IRQ_AUDIO_SHORT, "audio-short", IORESOURCE_IRQ,},
WARNING: line over 80 characters
#153: FILE: drivers/mfd/88pm860x-core.c:153:
+ {PM8607_IRQ_CHG_DONE, PM8607_IRQ_CHG_DONE, "charging done", IORESOURCE_IRQ,},
WARNING: line over 80 characters
#154: FILE: drivers/mfd/88pm860x-core.c:154:
+ {PM8607_IRQ_CHG_FAIL, PM8607_IRQ_CHG_FAIL, "charging timeout", IORESOURCE_IRQ,},
WARNING: line over 80 characters
#155: FILE: drivers/mfd/88pm860x-core.c:155:
+ {PM8607_IRQ_CHG_FAULT, PM8607_IRQ_CHG_FAULT, "charging fault", IORESOURCE_IRQ,},
WARNING: line over 80 characters
#156: FILE: drivers/mfd/88pm860x-core.c:156:
+ {PM8607_IRQ_GPADC1, PM8607_IRQ_GPADC1, "battery temperature", IORESOURCE_IRQ,},
WARNING: Avoid unnecessary line continuations
#571: FILE: drivers/mfd/88pm860x-core.c:571:
+ struct i2c_client *i2c = (chip->id == CHIP_PM8607) ? chip->client \
WARNING: line over 80 characters
#634: FILE: drivers/mfd/88pm860x-core.c:634:
+ ret = request_threaded_irq(chip->core_irq, NULL, pm860x_irq, flags | IRQF_ONESHOT,
WARNING: Unnecessary parentheses - maybe == should be = ?
#874: FILE: drivers/mfd/88pm860x-core.c:874:
+ if ((pdata == NULL))
WARNING: quoted string split across lines
#1001: FILE: drivers/mfd/88pm860x-core.c:1001:
+ dev_err(chip->dev, "Failed to detect Marvell 88PM8607. "
+ "Chip ID: %02x\n", ret);
WARNING: quoted string split across lines
#1124: FILE: drivers/mfd/88pm860x-core.c:1124:
+ dev_err(dev, "Not found \"marvell,88pm860x-slave-addr\" "
+ "property\n");
total: 0 errors, 11 warnings, 1281 lines checked
Signed-off-by: Lee Jones <[email protected]>
|
I'm new to Linux/Raspbian. smsc95xx.turbo_mode=N, vm.min_free_kbytes=2048 seems to be a good modification. Running it for 12h without an issue so far. Any reason why this is not of the standard build ? |
popcornmix
pushed a commit
that referenced
this issue
Jul 14, 2015
John Stultz reports an RCU splat on boot with ARM ipi trace events enabled. =============================== [ INFO: suspicious RCU usage. ] 4.1.0-rc7-00033-gb5bed2f #153 Not tainted ------------------------------- include/trace/events/ipi.h:68 suspicious rcu_dereference_check() usage! other info that might help us debug this: RCU used illegally from idle CPU! rcu_scheduler_active = 1, debug_locks = 0 RCU used illegally from extended quiescent state! no locks held by swapper/0/0. stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.1.0-rc7-00033-gb5bed2f #153 Hardware name: Qualcomm (Flattened Device Tree) [<c0216b08>] (unwind_backtrace) from [<c02136e8>] (show_stack+0x10/0x14) [<c02136e8>] (show_stack) from [<c075e678>] (dump_stack+0x70/0xbc) [<c075e678>] (dump_stack) from [<c0215a80>] (handle_IPI+0x428/0x604) [<c0215a80>] (handle_IPI) from [<c020942c>] (gic_handle_irq+0x54/0x5c) [<c020942c>] (gic_handle_irq) from [<c0766604>] (__irq_svc+0x44/0x7c) Exception stack(0xc09f3f48 to 0xc09f3f90) 3f40: 00000001 00000001 00000000 c09f73b8 c09f4528 c0a5de9c 3f60: c076b4f0 00000000 00000000 c09ef108 c0a5cec1 00000001 00000000 c09f3f90 3f80: c026bf60 c0210ab8 20000113 ffffffff [<c0766604>] (__irq_svc) from [<c0210ab8>] (arch_cpu_idle+0x20/0x3c) [<c0210ab8>] (arch_cpu_idle) from [<c02647f0>] (cpu_startup_entry+0x2c0/0x5dc) [<c02647f0>] (cpu_startup_entry) from [<c099bc1c>] (start_kernel+0x358/0x3c4) [<c099bc1c>] (start_kernel) from [<8020807c>] (0x8020807c) At this point in the IPI handling path we haven't called irq_enter() yet, so RCU doesn't know that we're about to exit idle and properly warns that we're using RCU from an idle CPU. Use trace_ipi_entry_rcuidle() instead of trace_ipi_entry() so that RCU is informed about our exit from idle. Fixes: 365ec7b ("ARM: add IPI tracepoints") Reported-by: John Stultz <[email protected]> Tested-by: John Stultz <[email protected]> Acked-by: Steven Rostedt <[email protected]> Reviewed-by: Paul E. McKenney <[email protected]> Signed-off-by: Stephen Boyd <[email protected]> Signed-off-by: Russell King <[email protected]>
davet321
pushed a commit
to davet321/rpi-linux
that referenced
this issue
Aug 4, 2015
commit 398f745 upstream. John Stultz reports an RCU splat on boot with ARM ipi trace events enabled. =============================== [ INFO: suspicious RCU usage. ] 4.1.0-rc7-00033-gb5bed2f raspberrypi#153 Not tainted ------------------------------- include/trace/events/ipi.h:68 suspicious rcu_dereference_check() usage! other info that might help us debug this: RCU used illegally from idle CPU! rcu_scheduler_active = 1, debug_locks = 0 RCU used illegally from extended quiescent state! no locks held by swapper/0/0. stack backtrace: CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.1.0-rc7-00033-gb5bed2f raspberrypi#153 Hardware name: Qualcomm (Flattened Device Tree) [<c0216b08>] (unwind_backtrace) from [<c02136e8>] (show_stack+0x10/0x14) [<c02136e8>] (show_stack) from [<c075e678>] (dump_stack+0x70/0xbc) [<c075e678>] (dump_stack) from [<c0215a80>] (handle_IPI+0x428/0x604) [<c0215a80>] (handle_IPI) from [<c020942c>] (gic_handle_irq+0x54/0x5c) [<c020942c>] (gic_handle_irq) from [<c0766604>] (__irq_svc+0x44/0x7c) Exception stack(0xc09f3f48 to 0xc09f3f90) 3f40: 00000001 00000001 00000000 c09f73b8 c09f4528 c0a5de9c 3f60: c076b4f0 00000000 00000000 c09ef108 c0a5cec1 00000001 00000000 c09f3f90 3f80: c026bf60 c0210ab8 20000113 ffffffff [<c0766604>] (__irq_svc) from [<c0210ab8>] (arch_cpu_idle+0x20/0x3c) [<c0210ab8>] (arch_cpu_idle) from [<c02647f0>] (cpu_startup_entry+0x2c0/0x5dc) [<c02647f0>] (cpu_startup_entry) from [<c099bc1c>] (start_kernel+0x358/0x3c4) [<c099bc1c>] (start_kernel) from [<8020807c>] (0x8020807c) At this point in the IPI handling path we haven't called irq_enter() yet, so RCU doesn't know that we're about to exit idle and properly warns that we're using RCU from an idle CPU. Use trace_ipi_entry_rcuidle() instead of trace_ipi_entry() so that RCU is informed about our exit from idle. Fixes: 365ec7b ("ARM: add IPI tracepoints") Reported-by: John Stultz <[email protected]> Tested-by: John Stultz <[email protected]> Acked-by: Steven Rostedt <[email protected]> Reviewed-by: Paul E. McKenney <[email protected]> Signed-off-by: Stephen Boyd <[email protected]> Signed-off-by: Russell King <[email protected]> Signed-off-by: Greg Kroah-Hartman <[email protected]>
popcornmix
pushed a commit
that referenced
this issue
Dec 21, 2015
…ting in skbuff clones
If we attach the sk to the skb from nfnetlink_rcv_batch(), then
netlink_skb_destructor() will underflow the socket receive memory
counter and we get warning splat when releasing the socket.
$ cat /proc/net/netlink
sk Eth Pid Groups Rmem Wmem Dump Locks Drops Inode
ffff8800ca903000 12 0 00000000 -54144 0 0 2 0 17942
^^^^^^
Rmem above shows an underflow.
And here below the warning splat:
[ 1363.815976] WARNING: CPU: 2 PID: 1356 at net/netlink/af_netlink.c:958 netlink_sock_destruct+0x80/0xb9()
[...]
[ 1363.816152] CPU: 2 PID: 1356 Comm: kworker/u16:1 Tainted: G W 4.4.0-rc1+ #153
[ 1363.816155] Hardware name: LENOVO 23259H1/23259H1, BIOS G2ET32WW (1.12 ) 05/30/2012
[ 1363.816160] Workqueue: netns cleanup_net
[ 1363.816163] 0000000000000000 ffff880119203dd0 ffffffff81240204 0000000000000000
[ 1363.816169] ffff880119203e08 ffffffff8104db4b ffffffff813d49a1 ffff8800ca771000
[ 1363.816174] ffffffff81a42b00 0000000000000000 ffff8800c0afe1e0 ffff880119203e18
[ 1363.816179] Call Trace:
[ 1363.816181] <IRQ> [<ffffffff81240204>] dump_stack+0x4e/0x79
[ 1363.816193] [<ffffffff8104db4b>] warn_slowpath_common+0x9a/0xb3
[ 1363.816197] [<ffffffff813d49a1>] ? netlink_sock_destruct+0x80/0xb9
skb->sk was only needed to lookup for the netns, however we don't need
this anymore since 633c9a8 ("netfilter: nfnetlink: avoid recurrent
netns lookups in call_batch") so this patch removes this manual socket
assignment to resolve this problem.
Reported-by: Arturo Borrero Gonzalez <[email protected]>
Reported-by: Ben Hutchings <[email protected]>
Signed-off-by: Pablo Neira Ayuso <[email protected]>
Tested-by: Arturo Borrero Gonzalez <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Running Transmission on a 512MB Pi, and with just two concurrent downloads, it's commonplace to see swapper page allocation failures:
also:
I've tried various settings for vm.min_free_kbytes, but I still see these swapper allocation failures even with 16384 and 32768.
When these errors occur, I typically have over 350MB free memory, according to free:
Once the page allocation error occurs, syslog and kern.log are spammed with 10's of thousands of kevent 2 messages being logged in a short space of time (almost 180K messages in 20 minutes, once to syslog and again to kern.log - see issue #154) - these message need to be rate limited.
Forum discussion link
The text was updated successfully, but these errors were encountered: