You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

George Joseph 98ac114678 CDR/CEL Custom Performance Improvements ... There is a LOT of work in this commit but the TL;DR is that it takes CEL processing from using 38% of the CPU instructions used by a call, which is more than that used by the call processing itself, down to less than 10% of the instructions. So here's the deal... cdr_custom, cdr_sqlite3_custom, cel_custom and cel_sqlite3_custom all shared one ugly trait...they all used ast_str_substitute_variables() or pbx_substitute_variables_helper() to resolve the dialplan functions used in their config files. Not only are they both extraordinarily expensive, they both require a dummy channel to be allocated and destroyed for each record written. For CDRs, that's not too bad because we only write one CDR per call. For CELs however, it's a disaster. As far as source code goes, the modules basically all did the same thing. Unfortunately, they did it badly. The config files simply contained long opaque strings which were intepreted by ast_str_substitute_variables() or pbx_substitute_variables_helper(), the very functions that ate all the instructions. This meant introducing a new "advanced" config format much like the advanced manager event filtering added to manager.conf in 2024. Fortunately however, if the legacy config was recognizable, we were able to parse it as an advanced config and gain the benefit. If not, then it goes the legacy, and very expensive, route. Given the commonality among the modules, instead of making the same improvements to 4 modules then trying to maintain them over time, a single module "res_cdrel_custom" was created that contains all of the common code. A few bonuses became possible in the process... * The cdr_custom and cel_custom modules now support JSON formatted output. * The cdr_sqlite_custom and cel_sqlite3_custom modules no longer have to share an Sqlite3 database. Summary of changes: A new module "res/res_cdrel_custom.c" has been created and the existing cdr_custom, cdr_sqlite3_custom, cel_custom and cel_sqlite3_custom modules are now just stubs that call the code in res_cdrel_custom. res_cdrel_custom contains: * A common configuration facility. * Getters for both CDR and CEL fields that share the same abstraction. * Formatters for all data types found in the ast_cdr and ast_event structures that share the same abstraction. * Common writers for the text file and database backends that, you guessed it, share the same abstraction. The result is that while there is certainly a net increase in the number of lines in the code base, most of it is in the configuration handling at load-time. The run-time instruction path length is now significanty shorter. ``` Scenario Instructions Latency ===================================================== CEL pre changes 38.49% 37.51% CEL Advanced 9.68% 6.06% CEL Legacy (auto-conv to adv) 9.95% 6.13% CEL Sqlite3 pre changes 39.41% 39.90% CEL Sqlite3 Advanced 25.68% 24.24% CEL Sqlite3 Legacy (auto conv) 25.88% 24.53% CDR pre changes 4.79% 2.95% CDR Advanced 0.79% 0.47% CDR Legacy (auto conv to adv) 0.86% 0.51% CDR Sqlite3 pre changes 4.47% 2.89% CEL Sqlite3 Advanced 2.16% 1.29% CEL Sqlite3 Legacy (auto conv) 2.19% 1.30% ``` Notes: * We only write one CDR per call but every little bit helps. * Sqlite3 still takes a fair amount of resources but the new config makes a decent improvement. * Legacy configs that we can't auto convert will still take the "pre changes" path. If you're interested in more implementation details, see the comments at the top of the res_cdrel_custom.c file. One minor fix to CEL is also included...Although TenantID was added to the ast_event structure, it was always rendered as an empty string. It's now properly rendered. UserNote: Significant performance improvements have been made to the cdr_custom, cdr_sqlite3_custom, cel_custom and cel_sqlite3_custom modules. See the new sample config files for those modules to see how to benefit from them.		5 months ago
..
Makefile	…
cdr_adaptive_odbc.c	…
cdr_beanstalkd.c	…
cdr_csv.c	…
cdr_custom.c	CDR/CEL Custom Performance Improvements	5 months ago
cdr_manager.c	…
cdr_odbc.c	…
cdr_pgsql.c	…
cdr_radius.c	…
cdr_sqlite3_custom.c	CDR/CEL Custom Performance Improvements	5 months ago
cdr_tds.c	…