Scale

The only limiting factor on FIB scale is the amount of memory allocated to each heap the FIB uses, and there are 4:

  • The IP4 heap

  • The IP6 heap

  • The main heap

  • The stats heap

IP4 Heap

The IPv4 heap is used to allocate the memory needed for the data-structures within which the IPv4 prefixes are stored. Each table, created by the user, i.e. with;

$ ip table add 1

or the default table, comprises 2 ip4_fib_t objects. The ‘non-forwarding’ ip4_fib_t contains all the entries in the table and, the ‘forwarding’ contains the entries that are matched against in the data-plane. The difference between the two sets are the entries that should not be matched in the data-plane. Each ip4_fib_t comprises an mtrie (for fast lookup in the data-plane) and a hash table per-prefix length (for lookup in the control plane).

To see the amount of memory consumed by the IPv4 tables use:

vpp# sh ip fib mem
ipv4-VRF:0 mtrie:333056 hash:3523
ipv4-VRF:1 mtrie:333056 hash:3523
totals: mtrie:666112 hash:7046 all:673158

Mtrie Mheap Usage: total: 32.06M, used: 662.44K, free: 31.42M, trimmable: 31.09M
    free chunks 3 free fastbin blks 0
    max total allocated 32.06M
no traced allocations

this output shows two ‘empty’ (i.e. no added routes) tables. Each mtrie uses about 150k of memory, so each table about 300k. the total heap usage statistics for the IP4 heap are shown at the end.

Below the output having added 1M, 2M and 4M routes respectively:

vpp# sh ip fib mem
ipv4-VRF:0 mtrie:335744 hash:4695
totals: mtrie:335744 hash:4695 all:340439

Mtrie Mheap Usage: total: 1.00G, used: 335.20K, free: 1023.74M, trimmable: 1023.72M
    free chunks 3 free fastbin blks 0
    max total allocated 1.00G
no traced allocations
vpp# sh ip fib mem
ipv4-VRF:0 mtrie:5414720 hash:41177579
totals: mtrie:5414720 hash:41177579 all:46592299

Mtrie Mheap Usage: total: 1.00G, used: 46.87M, free: 977.19M, trimmable: 955.93M
    free chunks 61 free fastbin blks 0
    max total allocated 1.00G
no traced allocations
vpp# sh ip fib mem
ipv4-VRF:0 mtrie:22452608 hash:168544508
totals: mtrie:22452608 hash:168544508 all:190997116

Mtrie Mheap Usage: total: 1.00G, used: 198.37M, free: 825.69M, trimmable: 748.24M
    free chunks 219 free fastbin blks 0
    max total allocated 1.00G
no traced allocations

VPP was started with a 1G IP4 heap.

IP6 Heap

The IPv6 heap is used to allocate the memory needed for the data-structure within which the IPv6 prefixes are stored. IPv6 also has the concept of forwarding and non-forwarding entries, however for IPv6 all the forwarding entries are stored in a single hash table (same goes for the non-forwarding). The key to the hash table includes the IPv6 table-id.

To see the amount of memory consumed by the IPv4 tables use:

vpp# sh ip6 fib mem
IPv6 Non-Forwarding Hash Table:
Hash table ip6 FIB non-fwding table
    7 active elements 7 active buckets
    1 free lists
    0 linear search buckets
    arena: base 7f2fe28bf000, next 803c0
           used 525248 b (0 Mbytes) of 33554432 b (32 Mbytes)

IPv6 Forwarding Hash Table:
Hash table ip6 FIB fwding table
    7 active elements 7 active buckets
    1 free lists
    0 linear search buckets
    arena: base 7f2fe48bf000, next 803c0
           used 525248 b (0 Mbytes) of 33554432 b (32 Mbytes)

as we scale to 128k IPv6 entries:

vpp# sh ip6 fib mem
IPv6 Non-Forwarding Hash Table:
Hash table ip6 FIB non-fwding table
    131079 active elements 32773 active buckets
    2 free lists
       [len 1] 2 free elts
    0 linear search buckets
    arena: base 7fed7a514000, next 4805c0
           used 4720064 b (4 Mbytes) of 1073741824 b (1024 Mbytes)

IPv6 Forwarding Hash Table:
Hash table ip6 FIB fwding table
    131079 active elements 32773 active buckets
    2 free lists
       [len 1] 2 free elts
    0 linear search buckets
    arena: base 7fedba514000, next 4805c0
           used 4720064 b (4 Mbytes) of 1073741824 b (1024 Mbytes)

and 256k:

vpp# sh ip6 fib mem
IPv6 Non-Forwarding Hash Table:
Hash table ip6 FIB non-fwding table
    262151 active elements 65536 active buckets
    2 free lists
       [len 1] 6 free elts
    0 linear search buckets
    arena: base 7fed7a514000, next 880840
           used 8915008 b (8 Mbytes) of 1073741824 b (1024 Mbytes)

IPv6 Forwarding Hash Table:
Hash table ip6 FIB fwding table
    262151 active elements 65536 active buckets
    2 free lists
       [len 1] 6 free elts
    0 linear search buckets
    arena: base 7fedba514000, next 880840
           used 8915008 b (8 Mbytes) of 1073741824 b (1024 Mbytes)

and 1M:

vpp# sh ip6 fib mem
IPv6 Non-Forwarding Hash Table:
Hash table ip6 FIB non-fwding table
    1048583 active elements 65536 active buckets
    4 free lists
       [len 1] 65533 free elts
       [len 2] 65531 free elts
       [len 4] 9 free elts
    0 linear search buckets
    arena: base 7fed7a514000, next 3882740
           used 59254592 b (56 Mbytes) of 1073741824 b (1024 Mbytes)

IPv6 Forwarding Hash Table:
Hash table ip6 FIB fwding table
    1048583 active elements 65536 active buckets
    4 free lists
       [len 1] 65533 free elts
       [len 2] 65531 free elts
       [len 4] 9 free elts
    0 linear search buckets
    arena: base 7fedba514000, next 3882740
           used 59254592 b (56 Mbytes) of 1073741824 b (1024 Mbytes)

as can be seen from the output the IPv6 heap in this case was scaled to 1GB and 1million prefixes has used 56MB of it.

Main Heap

The main heap is used to allocate objects that represent the FIB entries in the control and data plane (see The Control Plane and The Data Plane) such as fib_entry_t and load_balance_t. These come from the main heap because they are not protocol specific (i.e. they are used to represent either IPv4, IPv6 or MPLS entries).

With 1M prefixes allocated the memory usage is:

vpp# sh fib mem
FIB memory
 Tables:
            SAFI              Number     Bytes
        IPv4 unicast             1     33619968
        IPv6 unicast             2     118502784
            MPLS                 0         0
       IPv4 multicast            1       1175
       IPv6 multicast            1      525312
 Nodes:
            Name               Size  in-use /allocated   totals
            Entry               72   1048589/ 1048589    75498408/75498408
        Entry Source            40   1048589/ 1048589    41943560/41943560
    Entry Path-Extensions       76      0   /    0       0/0
       multicast-Entry         192      6   /    6       1152/1152
          Path-list             40     18   /    18      720/720
          uRPF-list             16     14   /    14      224/224
            Path                72     22   /    22      1584/1584
     Node-list elements         20   1048602/ 1048602    20972040/20972040
       Node-list heads          8      24   /    24      192/192

and with 2M

vpp# sh fib mem
FIB memory
 Tables:
            SAFI              Number     Bytes
        IPv4 unicast             1     33619968
        IPv6 unicast             2     252743040
            MPLS                 0         0
       IPv4 multicast            1       1175
       IPv6 multicast            1      525312
 Nodes:
            Name               Size  in-use /allocated   totals
            Entry               72   2097165/ 2097165    150995880/150995880
        Entry Source            40   2097165/ 2097165    83886600/83886600
    Entry Path-Extensions       76      0   /    0       0/0
       multicast-Entry         192      6   /    6       1152/1152
          Path-list             40     18   /    19      720/760
          uRPF-list             16     18   /    18      288/288
            Path                72     22   /    23      1584/1656
     Node-list elements         20   2097178/ 2097178    41943560/41943560
       Node-list heads          8      24   /    24      192/192

However, the situation is not a simple as that. All of the 1M prefixes added above were reachable via the same next-hop, so the path-list (and path) they use is shared. As prefixes are added that use different (sets of) next-hops, the number of path-lists and paths requires will increase.

Stats Heap

VPP collects statistics for each route. For each route VPP collects byte and packet counters for packets sent to the prefix (i.e. the route was matched in the data-plane) and packets sent via the prefix (i.e. the matching prefix is reachable through it - like a BGP peer). This requires 4 counters per route in the stats segment.

Below shows the size of the stats segment with 1M, 2M and 4M routes.

total: 1023.99M, used: 127.89M, free: 896.10M, trimmable: 830.94M
total: 1023.99M, used: 234.14M, free: 789.85M, trimmable: 668.15M
total: 1023.99M, used: 456.83M, free: 567.17M, trimmable: 388.91M

VPP was started with a 1G stats heap.