Could you add some information about what type of sets do you use, including the extensions? Also, do you add entries with IP addresses only, or hostnames and DNS lookups are involved as well.

Created attachment 299409 [details]
Selected sets in kernel

My firewall uses 12 ipset tables. Problematic tables uses hash:net. Except these 4 tables, other tables are much smaller. 1 table about 200 ip adresses and other below 20 entries.

# ipset list -t|grep -F "Type"
Type: hash:net
Type: hash:ip,mac
Type: hash:ip,mac
Type: hash:net
Type: hash:net
Type: hash:net
Type: hash:net
Type: hash:net
Type: hash:net
Type: hash:net
Type: hash:net
Type: hash:net

Source file with ip adresses, contains entries like these:
123.123.123.123
0.0.0.0/8

All entries are ip, not domain name.

In iptables, I am using standard extensions like:
- States (INVALID, ESTABLISHED etc)
- Ports (src and dst)
- IP (src and dst)
- Protocols (UDP, TCP, ICMP)
- MAC
- LOG/ACCEPT/DROP/REJECT
- match-set

Created attachment 299411 [details]
Function to add ip into ipset

This file (function.txt), contains function, what I am using to add ip into ipset. Names are in polish, but function is short, so you should understand, what this function do.
This function is run four times in the same time.

Is it enough for you to fix it? or you need more informations?

Could you send me in private a saved set content in order to be able to reproduce the issue more easily?


In your function you call "ipset add" for every element, which is not efficient. Better create the content of the set in saved format (simply "add setname element" lines in a file) and call "ipset restore < filename", which is way faster.

However, it does explain the slowdown in adding single entries one by one.

I can't send you all my sets, but I will send you as much as it's possible. I will try to send you more functions (without data), but it will be in next week (On next Tuesday or Wednesday I should be free to prepare it).
Problematic sets are lists from https://iplists.firehol.org (firehol_level1, firehol_level2, firehol_level3, and firehol_level4).

all_lists = firehol_level1 + firehol_level2 + firehol_level3 + firehol_level4
list1,list2,list3,list4 = all_lists/4
Files named list1, list2 etc. are added into ipset. 

I know, that I can save and load all ip adresses as one file, but I prepared these functions for smaller sets and it wasn't problem. Lists from firehol I started to use later.  For now, 10 minutes waiting is not a big problem, but 2h is a big problem, especially at laptop, where fan are loud.

Yes, it explains, why adding is slow, but it doesn't explain, why there is so big difference between old and new kernel. I am using the same data and functions.

Can you isolate the issue to ipset by removing the ipcalc call in your script and running it to see if it's still slower than you anticipate? I tried your script locally and ipcalc is the slowest part for me.

You might pre-filter your IPs using your ipcalc logic, and then feed that into a script that doesn't run the ipcalc check.

You are right. I checked it and problem is connected with ipcalc.
When I tried to use older version of ipcalc, my script worked well.
Probably, when I installed new version of kernel, in the meantime ipcalc was updated too and I thought that problem is connected with ipset.

They added IPv6 support into ipcalc and it cause performance decreasing.

PS. Do you know, how can I verify if item is correct ip address or network address? I have to replace ipcalc and use other tool.

Thanks for confirming. Can you not just stick with the older ipcalc version? You might also want to open an issue on ipcalc to let them know of this significant performance degradation: https://github.com/kjokjo/ipcalc