You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

George Joseph c948ce9651 sorcery/res_pjsip: Refactor for realtime performance ... There were a number of places in the res_pjsip stack that were getting all endpoints or all aors, and then filtering them locally. A good example is pjsip_options which, on startup, retrieves all endpoints, then the aors for those endpoints, then tests the aors to see if the qualify_frequency is > 0. One issue was that it never did anything with the endpoints other than retrieve the aors so we probably could have skipped a step and just retrieved all aors. But nevermind. This worked reasonably well with local config files but with a realtime backend and thousands of objects, this was a nightmare. The issue really boiled down to the fact that while realtime supports predicates that are passed to the database engine, the non-realtime sorcery backends didn't. They do now. The realtime engines have a scheme for doing simple comparisons. They take in an ast_variable (or list) for matching, and the name of each variable can contain an operator. For instance, a name of "qualify_frequency >" and a value of "0" would create a SQL predicate that looks like "where qualify_frequency > '0'". If there's no operator after the name, the engines add an '=' so a simple name of "qualify_frequency" and a value of "10" would return exact matches. The non-realtime backends decide whether to include an object in a result set by calling ast_sorcery_changeset_create on every object in the internal container. However, ast_sorcery_changeset_create only does exact string matches though so a name of "qualify_frequency >" and a value of "0" returns nothing because the literal "qualify_frequency >" doesn't match any name in the objset set. So, the real task was to create a generic string matcher that can take a left value, operator and a right value and perform the match. To that end, strings.c has a new ast_strings_match(left, operator, right) function. Left and right are the strings to operate on and the operator can be a string containing any of the following: = (or NULL or ""), !=, >, >=, <, <=, like or regex. If the operator is like or regex, the right string should be a %-pattern or a regex expression. If both left and right can be converted to float, then a numeric comparison is performed, otherwise a string comparison is performed. To use this new function on ast_variables, 2 new functions were added to config.c. One that compares 2 ast_variables, and one that compares 2 ast_variable lists. The former is useful when you want to compare 2 ast_variables that happen to be in a list but don't want to traverse the list. The latter will traverse the right list and return true if all the variables in it match the left list. Now, the backends' fields_cmp functions call ast_variable_lists_match instead of ast_sorcery_changeset_create and they can now process the same syntax as the realtime engines. The realtime backend just passes the variable list unaltered to the engine. The only gotcha is that there's no common realtime engine support for regex so that's been noted in the api docs for ast_sorcery_retrieve_by_fields. Only one more change to sorcery was done... A new config flag "allow_unqualified_fetch" was added to reg_sorcery_realtime. "no": ignore fetches if no predicate fields were supplied. "error": same as no but emit an error. (good for testing) "yes": allow (the default); "warn": allow but emit a warning. (good for testing) Now on to res_pjsip... pjsip_options was modified to retrieve aors with qualify_frequency > 0 rather than all endpoints then all aors. Not only was this a big improvement in realtime retrieval but even for config files there's an improvement because we're not going through endpoints anymore. res_pjsip_mwi was modified to retieve only endpoints with something in the mailboxes field instead of all endpoints then testing mailboxes. res_pjsip_registrar_expire was completely refactored. It was retrieving all contacts then setting up scheduler entries to check for expiration. Now, it's a single thread (like keepalive) that periodically retrieves only contacts whose expiration time is < now and deletes them. A new contact_expiration_check_interval was added to global with a default of 30 seconds. Ross Beer reports that with this patch, his Asterisk startup time dropped from around an hour to under 30 seconds. There are still objects that can't be filtered at the database like identifies, transports, and registrations. These are not going to be anywhere near as numerous as endpoints, aors, auths, contacts however. Back to allow_unqualified_fetch. If this is set to yes and you have a very large number of objects in the database, the pjsip CLI commands will attempt to retrive ALL of them if not qualified with a LIKE. Worse, if you type "pjsip show endpoint <tab>" guess what's going to happen? :) Having a cache helps but all the objects will have to be retrieved at least once to fill the cache. Setting allow_unqualified_fetch=no prevents the mass retrieve and should be used on endpoints, auths, aors, and contacts. It should NOT be used for identifies, registrations and transports since these MUST be retrieved in bulk. Example sorcery.conf: [res_pjsip] endpoint=config,pjsip.conf,criteria=type=endpoint endpoint=realtime,ps_endpoints,allow_unqualified_fetch=error ASTERISK-25826 #close Reported-by: Ross Beer Tested-by: Ross Beer Change-Id: Id2691e447db90892890036e663aaf907b2dc1c67		9 years ago
..
Makefile	Title update	13 years ago
test_abstract_jb.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_acl.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_ami_security_events.sh	Add an API for reporting security events, and a security event logging module.	16 years ago
test_amihooks.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_aoc.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_app.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_ari.c	doxygen: Fix doxygen errors	10 years ago
test_ari_model.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_ast_format_str_reduce.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_astobj2.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_astobj2_thrash.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_astobj2_weaken.c	Astobj2: Add ao2_weakproxy_ref_object function.	10 years ago
test_bucket.c	bucket: Add clone/staleness operations for ast_bucket/ast_bucket_file	10 years ago
test_callerid.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_cdr.c	Unit tests: Fix unit test description strings.	10 years ago
test_cel.c	Unit tests: Fix unit test description strings.	10 years ago
test_channel_feature_hooks.c	Unit tests: Fix unit test description strings.	10 years ago
test_config.c	sorcery/res_pjsip: Refactor for realtime performance	9 years ago
test_core_codec.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_core_format.c	main/format: Add an API call for retrieving format attributes	10 years ago
test_db.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_devicestate.c	tests/test_devicestate: Add additional tests for the device state API	10 years ago
test_dlinklists.c	logging: Remove/fix some message annoyances	9 years ago
test_dns.c	Merge "Replaces clock_gettime() with ast_tsnow()"	10 years ago
test_dns_naptr.c	Unit tests: Fix more unit test description strings.	10 years ago
test_dns_query_set.c	Replaces clock_gettime() with ast_tsnow()	10 years ago
test_dns_recurring.c	Replaces clock_gettime() with ast_tsnow()	10 years ago
test_dns_srv.c	Unit tests: Fix more unit test description strings.	10 years ago
test_endpoints.c	doxygen: Fix doxygen errors	10 years ago
test_event.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_expr.c	Unit tests: Fix unit test description strings.	10 years ago
test_format_cache.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_format_cap.c	format_cap: Don't append the 'none' format when appending all.	10 years ago
test_func_file.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_gosub.c	Fix unsafe uses of ast_context pointers.	10 years ago
test_hashtab_thrash.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_heap.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_http_media_cache.c	tests/test_http_media_cache: Fix file descriptor leak in test.	9 years ago
test_jitterbuf.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_json.c	doxygen: Fix doxygen errors	10 years ago
test_linkedlists.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_locale.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_logger.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_media_cache.c	media cache: Add a core API and facade for a backend agnostic media cache	10 years ago
test_message.c	Unit tests: Fix unit test description strings.	10 years ago
test_netsock2.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_optional_api.c	doxygen: Fix doxygen errors	10 years ago
test_pbx.c	Fix unsafe uses of ast_context pointers.	10 years ago
test_poll.c	Unit tests: Fix unit test description strings.	10 years ago
test_res_stasis.c	doxygen: Fix doxygen errors	10 years ago
test_sched.c	sched.c: Ensure oldest expiring entry runs first.	9 years ago
test_scoped_lock.c	ao2_iterator: Mini-audit of the ao2_iterator loops in the new code files.	12 years ago
test_security_events.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_skel.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_sorcery.c	sorcery: Add support for object staleness	10 years ago
test_sorcery_astdb.c	sorcery/res_pjsip: Refactor for realtime performance	9 years ago
test_sorcery_memory_cache_thrash.c	tests/test_sorcery_memory_cache_thrash: Improve termination process.	9 years ago
test_sorcery_realtime.c	sorcery/res_pjsip: Refactor for realtime performance	9 years ago
test_stasis.c	Unit tests: Fix unit test description strings.	10 years ago
test_stasis_channels.c	doxygen: Fix doxygen errors	10 years ago
test_stasis_endpoints.c	tests/test_stasis_endpoints: Remove expected duplicate events	10 years ago
test_stringfields.c	clang compiler warnings: Fix various warnings for tests	10 years ago
test_strings.c	sorcery/res_pjsip: Refactor for realtime performance	9 years ago
test_substitution.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_taskprocessor.c	Multiple revisions 399887,400138,400178,400180-400181	12 years ago
test_threadpool.c	Fix failing threadpool_auto_increment test.	9 years ago
test_time.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_uri.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_utils.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_uuid.c	Add uuid wrapper API call ast_uuid_generate_str().	12 years ago
test_vector.c	vector: Add REMOVE, ADD_SORTED and RESET macros	10 years ago
test_voicemail_api.c	Unit tests: Fix unit test description strings.	10 years ago
test_websocket_client.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago
test_xml_escape.c	git migration: Refactor the ASTERISK_FILE_VERSION macro	10 years ago