Skip to content

Improve networks_file storage and load/query algorithm for IPv6 networks #660

Open
@rodecker

Description

Description
I have some hosts that cannot receive full routing tables via BGP, so I wish to load them via a networks file. I've noticed a large performance difference between loading the IPv4 and IPv6 full routing tables from file. Loading the IPv6 table takes a very long time.

Build and config

pmacctd 1.7.9-git (20230121-0 (b5637b0))`
./configure --enable-jansson --enable-kafka
! pmacctd.conf
daemonize: false
pre_tag_map: /etc/pmacct/pretag.map
networks_file: /etc/pmacct/networks.lst
networks_file_no_lpm: true
!
!
aggregate: label, sampling_rate, in_iface, out_iface, src_mac, dst_mac, src_as, dst_as, src_net, dst_net, src_host, dst_host, etype, timestamp_arrival
!
! pcap capturing config
pcap_ifindex: map
pcap_interfaces_map: /etc/pmacct/iface.map
sampling_rate: 10 
pmacctd_renormalize: true
pmacctd_as: longest
pmacctd_net: longest
!
! export config
plugins: print
print_refresh_time: 60
print_output: json
!
timestamps_rfc3339: true
timestamps_secs: true

Steps to reproduce

Download IPv4 prefix list
# whois -h riswhois.ripe.net -- -F -M 0/0 | egrep -v "^%" | egrep -v "^$" | awk '{ print $1","$2; }' > networks.lst
# wc -l networks.lst
1093778 networks.lst

Start pmacctd
# /usr/local/sbin/pmacctd -f /etc/pmacct/pmacctd.conf -d

Wait until the loading is done
WARN ( default_print/print ): no print_output_file and no print_output_lock_file defined.

The load time is +/- 1 minute. During this time v4 nh: debug messages are continuously written to the screen while prefixes are being loaded.

Download IPv6 prefix list
# whois -h riswhois.ripe.net -- -F -M ::/0 | egrep -v "^%" | egrep -v "^$" | awk '{ print $1","$2; }' > networks.lst
# wc -l networks.lst
224898 networks.lst

Start pmacctd
# /usr/local/sbin/pmacctd -f /etc/pmacct/pmacctd.conf -d

Wait until the loading is done.
WARN ( default_print/print ): no print_output_file and no print_output_lock_file defined.

The load time is +/- 13 minutes. Most of this time no output is written to the screen and the pmacctd process is spinning at 99.9% CPU. The long processing time is in step 4a, bulding hierarchies: https://github.com/pmacct/pmacct/blob/master/src/net_aggr.c#L1532-L1559

Is this expected?

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions