path: root/mm/migrate.c
diff options
authorMinchan Kim <>2021-05-04 18:36:54 -0700
committerLinus Torvalds <>2021-05-05 11:27:24 -0700
commitd479960e44f27e0e52ba31b21740b703c538027c (patch)
tree779dfc5da9ab1ed52fbbca1dfec034ef769f81af /mm/migrate.c
parent06dac2f467fe9269a433aa5056dd2ee1d20475e9 (diff)
mm: disable LRU pagevec during the migration temporarily
LRU pagevec holds refcount of pages until the pagevec are drained. It could prevent migration since the refcount of the page is greater than the expection in migration logic. To mitigate the issue, callers of migrate_pages drains LRU pagevec via migrate_prep or lru_add_drain_all before migrate_pages call. However, it's not enough because pages coming into pagevec after the draining call still could stay at the pagevec so it could keep preventing page migration. Since some callers of migrate_pages have retrial logic with LRU draining, the page would migrate at next trail but it is still fragile in that it doesn't close the fundamental race between upcoming LRU pages into pagvec and migration so the migration failure could cause contiguous memory allocation failure in the end. To close the race, this patch disables lru caches(i.e, pagevec) during ongoing migration until migrate is done. Since it's really hard to reproduce, I measured how many times migrate_pages retried with force mode(it is about a fallback to a sync migration) with below debug code. int migrate_pages(struct list_head *from, new_page_t get_new_page, .. .. if (rc && reason == MR_CONTIG_RANGE && pass > 2) { printk(KERN_ERR, "pfn 0x%lx reason %d", page_to_pfn(page), rc); dump_page(page, "fail to migrate"); } The test was repeating android apps launching with cma allocation in background every five seconds. Total cma allocation count was about 500 during the testing. With this patch, the dump_page count was reduced from 400 to 30. The new interface is also useful for memory hotplug which currently drains lru pcp caches after each migration failure. This is rather suboptimal as it has to disrupt others running during the operation. With the new interface the operation happens only once. This is also in line with pcp allocator cache which are disabled for the offlining as well. Link: Signed-off-by: Minchan Kim <> Reviewed-by: Chris Goldsworthy <> Acked-by: Michal Hocko <> Cc: John Dias <> Cc: Suren Baghdasaryan <> Cc: Matthew Wilcox <> Cc: David Hildenbrand <> Cc: Vlastimil Babka <> Cc: Oliver Sang <> Signed-off-by: Andrew Morton <> Signed-off-by: Linus Torvalds <>
Diffstat (limited to 'mm/migrate.c')
1 files changed, 7 insertions, 4 deletions
diff --git a/mm/migrate.c b/mm/migrate.c
index 47df0df8f21a..5b09567dc293 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -66,11 +66,13 @@ void migrate_prep(void)
* Clear the LRU lists so pages can be isolated.
- * Note that pages may be moved off the LRU after we have
- * drained them. Those pages will fail to migrate like other
- * pages that may be busy.
- lru_add_drain_all();
+ lru_cache_disable();
+void migrate_finish(void)
+ lru_cache_enable();
/* Do the necessary work of migrate_prep but not if it involves other CPUs */
@@ -1838,6 +1840,7 @@ out_flush:
if (err >= 0)
err = err1;
+ migrate_finish();
return err;