Refilling the PerCPU Cache

The workload is heavier when there are no more objects in the per-CPU cache. The refill operations needed in this situation are located in cache_alloc_refill, which is invoked when the allocation cannot be satisfied directly from the per-CPU cache.

The kernel must now find array_cache->batchcount unused objects to refill the per-CPU cache by first scanning the list of all partially free slabs (slabs_partial) and then taking all free objects one after another by slab_get_obj until no more objects are free in the relevant slab. The kernel then performs the same procedure on all other slabs in the slabs_partial list. If this finds the desired number of objects, the kernel iterates over the slabs_free list of all unused slabs. When objects are taken from a slab, the kernel must also ensure that it places them on the correct slab list (slabs_full or slabs_partial, depending on whether the slab was totally emptied or still contains some objects). The above is implemented by the following code:

mm/slab.c static void *cache_alloc_refill(kmem_cache_t *cachep, gfp_t flags) {

/* Select list from which slabs are to be taken (first slabs_partial, then slabs_free) */

slabp = list_entry(entry, struct slab, list); while (slabp->inuse < cachep->num && batchcount--) { /* get obj pointer */

ac->entry[ac->avail++] = slab_get_obj(cachep, slabp, node);

check_slabp(cachep, slabp);


list_add(&slabp->list, &l3->slabs_full);

else list_add(&slabp->list, &!3->slabs_partial);

The key to removing one slab element after another is in slab_get_obj: mm/slab.c static void *slab_get_obj(struct kmem_cache *cachep, struct slab *slabp, int nodeid)

void *objp = index_to_obj(cachep, slabp, slabp->free); kmem_bufctl_t next;

next = slab_bufctl(slabp)[slabp->free]; slabp->free = next;

return objp;

Recall from Figure 3-47 that the kernel uses an interesting system to keep track of free entries: The index of the free object that is currently under consideration is stored in slabp->free, and the index of the next free object, is kept in the management array.

Obtaining the object that belongs to a given index is a matter of some simple pointer manipulation performed in index_to_obj. slab_bufctl is a macro that yields a pointer to the kmem_bufctl array after slabp.

Let us return to cache_alloc_grow. If no free object is found although all slabs have been scanned, the cache must be enlarged using cache_grow. This is a costly operation examined in the next section.

Continue reading here: Growing the Cache

Was this article helpful?

0 0