Slow route server config generation with v6.0.0.

[ichilton]

ISSUE TYPE

Bug Report

OS

Ubuntu 20.04

VERSION

define( 'APPLICATION_VERSION', '6.0.0' );
define( 'APPLICATION_VERDATE', '2021071400' );

ENVIRONMENT

ixpmanager@portal2:/srv/ixpmanager$ php -v
PHP 8.0.9 (cli) (built: Jul 30 2021 13:03:39) ( NTS )
Copyright (c) The PHP Group
Zend Engine v4.0.9, Copyright (c) Zend Technologies
    with Zend OPcache v8.0.9, Copyright (c), by Zend Technologies
ixpmanager@portal2:/srv/ixpmanager$ dpkg -l | grep php
ii  libapache2-mod-php8.0          8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        server-side, HTML-embedded scripting language (Apache 2 module)
ii  php-common                     2:84+ubuntu20.04.1+deb.sury.org+1                                   all          Common files for PHP packages
ii  php-composer-ca-bundle         1.2.6-1                                                             all          utility library to find a path to the system CA bundle
ii  php-composer-semver            1.5.1-1                                                             all          utilities, version constraint parsing and validation
ii  php-composer-spdx-licenses     1.5.3-1                                                             all          SPDX licenses list and validation library
ii  php-composer-xdebug-handler    1.4.0-1                                                             all          Restarts a process without Xdebug
ii  php-ds                         1.3.0-6+ubuntu20.04.1+deb.sury.org+1                                amd64        PHP extension providing efficient data structures for PHP 7
ii  php-json-schema                5.2.9-1                                                             all          implementation of JSON schema
ii  php-memcache                   8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-1+ubuntu20.04.1+deb.sury.org+2 amd64        memcache extension module for PHP
ii  php-memcached                  3.1.5+2.2.0-9+ubuntu20.04.1+deb.sury.org+1                          amd64        memcached extension module for PHP, uses libmemcached
ii  php-psr-container              1.0.0-2                                                             all          Common Container Interface (PHP FIG PSR-11)
ii  php-psr-log                    1.1.2-1                                                             all          common interface for logging libraries
ii  php-rrd                        2.0.3+1.1.3-1+ubuntu20.04.1+deb.sury.org+1                          amd64        PHP bindings to rrd tool system
ii  php-symfony-console            4.3.8+dfsg-1ubuntu1                                                 all          run tasks from the command line
ii  php-symfony-filesystem         4.3.8+dfsg-1ubuntu1                                                 all          basic filesystem utilities
ii  php-symfony-finder             4.3.8+dfsg-1ubuntu1                                                 all          find files and directories
ii  php-symfony-process            4.3.8+dfsg-1ubuntu1                                                 all          execute commands in sub-processes
ii  php-symfony-service-contracts  1.1.8-1                                                             all          Generic abstractions related to writing services
ii  php-yaml                       2.2.1+2.1.0+2.0.4+1.3.2-6+ubuntu20.04.1+deb.sury.org+1              amd64        YAML-1.1 parser and emitter for PHP
ii  php8.0                         8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                all          server-side, HTML-embedded scripting language (metapackage)
ii  php8.0-bcmath                  8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        Bcmath module for PHP
ii  php8.0-cgi                     8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        server-side, HTML-embedded scripting language (CGI binary)
ii  php8.0-cli                     8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        command-line interpreter for the PHP scripting language
ii  php8.0-common                  8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        documentation, examples and common module for PHP
ii  php8.0-curl                    8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        CURL module for PHP
ii  php8.0-ds                      1.3.0-6+ubuntu20.04.1+deb.sury.org+1                                amd64        PHP extension providing efficient data structures for PHP 7
ii  php8.0-fpm                     8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        server-side, HTML-embedded scripting language (FPM-CGI binary)
ii  php8.0-gd                      8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        GD module for PHP
ii  php8.0-igbinary                3.2.4+2.0.8-1+ubuntu20.04.1+deb.sury.org+1                          amd64        igbinary PHP serializer
ii  php8.0-intl                    8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        Internationalisation module for PHP
ii  php8.0-mbstring                8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        MBSTRING module for PHP
ii  php8.0-memcache                8.0+4.0.5.2+3.0.9~20170802.e702b5f9+-1+ubuntu20.04.1+deb.sury.org+2 amd64        memcache extension module for PHP
ii  php8.0-memcached               3.1.5+2.2.0-9+ubuntu20.04.1+deb.sury.org+1                          amd64        memcached extension module for PHP, uses libmemcached
ii  php8.0-msgpack                 2.1.2+0.5.7-6+ubuntu20.04.1+deb.sury.org+1                          amd64        PHP extension for interfacing with MessagePack
ii  php8.0-mysql                   8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        MySQL module for PHP
ii  php8.0-opcache                 8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        Zend OpCache module for PHP
ii  php8.0-readline                8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        readline module for PHP
ii  php8.0-rrd                     2.0.3+1.1.3-1+ubuntu20.04.1+deb.sury.org+1                          amd64        PHP bindings to rrd tool system
ii  php8.0-snmp                    8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        SNMP module for PHP
ii  php8.0-xml                     8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        DOM, SimpleXML, XML, and XSL module for PHP
ii  php8.0-yaml                    2.2.1+2.1.0+2.0.4+1.3.2-6+ubuntu20.04.1+deb.sury.org+1              amd64        YAML-1.1 parser and emitter for PHP
ii  php8.0-zip                     8.0.9-1+ubuntu20.04.1+deb.sury.org+1                                amd64        Zip module for PHP

CONFIGURATION

APP_DEBUG=false
APP_URL="https://portal.mydomain.net"
APP_TIMEZONE="Europe/London"
APP_LOG="single"
APP_LOG_LEVEL=debug
IXP_MULTIIXP_ENABLED=false
IXP_RESELLER_ENABLED=true
IXP_RESELLER_RESOLD_BILLING=false
IXP_AS112_UI_ACTIVE=false
IXP_FE_FRONTEND_DISABLED_LOGO=false
IXP_FE_CUSTOMER_BILLING_UPDATES_NOTIFY="REDACTED"
IXP_FE_FRONTEND_DISABLED_RS_PREFIXES=true
IXP_FE_FRONTEND_DISABLED_NETINFO=true
IXP_FE_FRONTEND_DISABLED_CUSTKIT=true
IXP_FE_FRONTEND_DISABLED_CONSOLE=true
IXP_FE_FRONTEND_DISABLED_LOGO=false
IXP_FE_FRONTEND_DISABLED_LOOKING_GLASS=false
IXP_FE_FRONTEND_DISABLED_FILTERED_PREFIXES=false
IXP_FE_BETA_CORE_BUNDLES=true
IXP_FE_VLANINTERFACES_HOSTNAME_REQUIRED=false
GRAPHER_BACKENDS="mrtg|sflow"
GRAPHER_CACHE_ENABLED=true
GRAPHER_BACKEND_MRTG_DBTYPE="log"
GRAPHER_BACKEND_MRTG_WORKDIR="/srv/mrtg"
GRAPHER_BACKEND_MRTG_LOGDIR="REDACTED"
GRAPHER_BACKEND_SFLOW_ENABLED=true
GRAPHER_BACKEND_SFLOW_ROOT="REDACTED"
GRAPHER_BACKEND_SMOKEPING_ENABLED=false
IXP_API_JSONEXPORTSCHEMA_PUBLIC=true
VIEW_SKIN="REDACTED"
VIEW_SMARTY_SKIN="REDACTED"
CACHE_DRIVER=memcached
DOCTRINE_PROXY_AUTOGENERATE=false
DOCTRINE_CACHE=array
DOCTRINE_CACHE_NAMESPACE=IXPMANAGERNAMESPACE
[email protected]
[email protected]
IXP_IRRDB_BGPQ3_PATH=/usr/bin/bgpq3
IXP_IRRDB_MIN_V4_SUBNET_SIZE=24
IXP_IRRDB_MIN_V6_SUBNET_SIZE=48
IXP_RPKI_RTR1_HOST=REDACTED
IXP_RPKI_RTR1_PORT=3323
IXP_RPKI_RTR2_HOST=REDACTED
IXP_RPKI_RTR2_PORT=3323
AUTH_PEERINGDB_ENABLED=true
PEERINGDB_OAUTH_CLIENT_ID="REDACTED"
PEERINGDB_OAUTH_CLIENT_SECRET="REDACTED"
PEERINGDB_OAUTH_REDIRECT="REDACTED"
AUTH_PEERINGDB_PRIVS=2
AUTH_ZF1_PWHASH="bcrypt"
SESSION_DRIVER=file

SUMMARY

Route server configuration is very slow compared to our old host on 5.8.0.

We have set up a new host with Ubuntu 20.04, IXP Manager v6.0.0, PHP 8.0 (FPM), Nginx.

Old host is the same spec VM, on the same VM cluster, but running Ubuntu 18.04, IXP Manager v5.8.0, PHP 7.3, Apache.

Same MySQL server, using a copy of the production database (with migrations ran for v6.0.0).

STEPS TO REPRODUCE

time curl -H "X-IXP-Manager-API-Key: mykey" https://portal.mydomain.net/api/v4/router/gen-config/rs3-ipv4

EXPECTED RESULTS

Similar results to the old host/version.

ACTUAL RESULTS

v5.8.0:

v4: 1m11.659s, 0m55.893s, 1m11.338s, 0m51.738s
v6: 0m33.376s, 0m32.611s, 0m33.523s, 0m34.540s

v6.0.0:

v4: 2m56.270s, 2m53.221s, 2m44.164s, 2m47.211s
v6: 0m55.845s, 0m54.098s, 0m58.139s, 0m59.906s

IMPORTANCE

RELEVANT LOGS

[2021-08-18 11:57:47] production.INFO: Generated router configuration for rs3-ipv4 and used 1335466608 bytes (1542991872 real) of memory.
[2021-08-18 12:00:41] production.INFO: Generated router configuration for rs3-ipv4 and used 1335466576 bytes (1542991872 real) of memory.
[2021-08-18 12:03:25] production.INFO: Generated router configuration for rs3-ipv4 and used 1341154136 bytes (1220030464 real) of memory.
[2021-08-18 12:06:13] production.INFO: Generated router configuration for rs3-ipv4 and used 1341154136 bytes (892874752 real) of memory.

[barryo]

Thanks for the detailed report @ichilton.

It looks like the queries did get less optimal in v6.0.0. This is a combination of Doctrine ORM vs Laravel Eloquent; their hydration methods (or at least the way we were doing it); and the actual query structure itself.

Regardless, we've known for sometime that the generation time was quite sub-optimal and looking at that has been on the list.

The tl;dr is that in commit b852c1f we've gotten the speed / usage down as follows:

Iteration	Generation Time	Memory Usage (Real)
v6.0.0	99.5 secs	1,362 MB (1,551 MB)
1st Pass - Query Optimisation	32.0 secs	501 MB (576 MB)
2nd Pass - Add Caching	1.3 secs	210 MB (212 MB)

NB: this is on my development laptop with local MySQL and webserver. YMMV for production environments but the speed up should be proportional.

This is now live and working on release-v6 and once we've watched it for a couple days we'll stamp a new release.

There is a further optimisation that could be made which would probably reduce the memory usage but push up the generation time - I think the above is okay for now. Anyway, route servers are top-tier production services so we'll iterate in baby steps.

Caching Detail for Those Interested

When we refresh the database's IRRDB records for prefixes and ASNs (per these instructions), we now also cache those arrays on a per-customer and per-protocol basis to the file-backed caching system. We specifically use the file-based system here to not push memcached / redis memory up for these large data sets. The goal is not lightening fast access but rather to avoid querying the database.

The caching itself is not done at the point where bgpq3 returns the data. This would be the ideal place to do it but that would yield randomly ordered data. Now, naturally, the Bird BGP daemon doesn't need this ordered but as engineers it's useful: we often diff generated configs when iterating changes - unordered prefix sets cause diffs with tens of thousands of changes; our continuous integration static files would also fail; and lastly when actually looking through the config to solve issues, ordering is useful.

So. the IRRDB update scripts will run longer because of this. These are background processes that run automatically so this is a compromise worth living with. For now at least.

[barryo]

Now released: https://github.com/inex/IXP-Manager/releases/tag/v6.1.0

[ichilton]

Hi @barryo,

Thank you for this! - I have tested this and it is indeed really fast now!

Running the above tests again on 6.1.0, I now see:

v4: 0m6.559s
v6: 0m2.594s

Thank you!

Ian