14 Commits (38d081c9f6578e749ecc2b9cc2912bed72e4b9e2)

Author	SHA1	Message	Date
Sean Bright	a4b291029f	astdb: Improve prefix searches in astdb ... Using the LIKE operator requires a full table scan of 'astdb', whereas a comparison operation is able to use the primary key index. This patch adds a new function to the AstDB API for quick prefix matches and updates res_sorcery_astdb to utilize it. This showed substantial performance improvement in my test environment. Related to ASTERISK~26806, but does not completely resolve it. Change-Id: I7d37f9ba2aea139dabf2ca72d31fbe34bd9b2fa1	8 years ago
Sean Bright	cf062303e3	sorcery: Add ast_sorcery_retrieve_by_prefix() ... Some consumers of the sorcery API use ast_sorcery_retrieve_by_regex only so that they can anchor the potential match as a prefix and not because they truly need regular expressions. Rather than using regular expressions for simple prefix lookups, add a new operation - ast_sorcery_retrieve_by_prefix - that does them. Patches against 13 and 15 have a compatibility layer needed to maintain ABI that is not needed in master. Change-Id: I56f4e20ba1154bd52281f995c27a429a854f6a79	8 years ago
Joshua Colp	ddcf983e39	res_sorcery_astdb: Filter fields to only the registered ones. ... This change introduces the same filtering that is done in res_sorcery_realtime to the res_sorcery_astdb module. This allows persisted sorcery objects that may contain unknown fields to still be read in from the AstDB and used. This is particularly useful when switching between different versions of Asterisk that may have introduced additional fields. ASTERISK-26014 #close Change-Id: Ib655130485a3ccfd635b7ed5546010ca14690fb2	9 years ago
Alexei Gradinari	9c2032240e	res_pjsip: improve realtime performance ... This patch modified pjsip_options to retrieve only permament contacts for aor if the qualify_frequency is > 0 and persisted contacts if the qualify_frequency is > 0. This patch also fixed a bug in res_sorcery_astdb. res_sorcery_astdb doesn't save object data retrived from astdb. ASTERISK-25826 Change-Id: I1831fa46c4578eae5a3e574ee3362fddf08a1f05	9 years ago
George Joseph	5aa5c49413	sorcery/res_pjsip: Refactor for realtime performance ... There were a number of places in the res_pjsip stack that were getting all endpoints or all aors, and then filtering them locally. A good example is pjsip_options which, on startup, retrieves all endpoints, then the aors for those endpoints, then tests the aors to see if the qualify_frequency is > 0. One issue was that it never did anything with the endpoints other than retrieve the aors so we probably could have skipped a step and just retrieved all aors. But nevermind. This worked reasonably well with local config files but with a realtime backend and thousands of objects, this was a nightmare. The issue really boiled down to the fact that while realtime supports predicates that are passed to the database engine, the non-realtime sorcery backends didn't. They do now. The realtime engines have a scheme for doing simple comparisons. They take in an ast_variable (or list) for matching, and the name of each variable can contain an operator. For instance, a name of "qualify_frequency >" and a value of "0" would create a SQL predicate that looks like "where qualify_frequency > '0'". If there's no operator after the name, the engines add an '=' so a simple name of "qualify_frequency" and a value of "10" would return exact matches. The non-realtime backends decide whether to include an object in a result set by calling ast_sorcery_changeset_create on every object in the internal container. However, ast_sorcery_changeset_create only does exact string matches though so a name of "qualify_frequency >" and a value of "0" returns nothing because the literal "qualify_frequency >" doesn't match any name in the objset set. So, the real task was to create a generic string matcher that can take a left value, operator and a right value and perform the match. To that end, strings.c has a new ast_strings_match(left, operator, right) function. Left and right are the strings to operate on and the operator can be a string containing any of the following: = (or NULL or ""), !=, >, >=, <, <=, like or regex. If the operator is like or regex, the right string should be a %-pattern or a regex expression. If both left and right can be converted to float, then a numeric comparison is performed, otherwise a string comparison is performed. To use this new function on ast_variables, 2 new functions were added to config.c. One that compares 2 ast_variables, and one that compares 2 ast_variable lists. The former is useful when you want to compare 2 ast_variables that happen to be in a list but don't want to traverse the list. The latter will traverse the right list and return true if all the variables in it match the left list. Now, the backends' fields_cmp functions call ast_variable_lists_match instead of ast_sorcery_changeset_create and they can now process the same syntax as the realtime engines. The realtime backend just passes the variable list unaltered to the engine. The only gotcha is that there's no common realtime engine support for regex so that's been noted in the api docs for ast_sorcery_retrieve_by_fields. Only one more change to sorcery was done... A new config flag "allow_unqualified_fetch" was added to reg_sorcery_realtime. "no": ignore fetches if no predicate fields were supplied. "error": same as no but emit an error. (good for testing) "yes": allow (the default); "warn": allow but emit a warning. (good for testing) Now on to res_pjsip... pjsip_options was modified to retrieve aors with qualify_frequency > 0 rather than all endpoints then all aors. Not only was this a big improvement in realtime retrieval but even for config files there's an improvement because we're not going through endpoints anymore. res_pjsip_mwi was modified to retieve only endpoints with something in the mailboxes field instead of all endpoints then testing mailboxes. res_pjsip_registrar_expire was completely refactored. It was retrieving all contacts then setting up scheduler entries to check for expiration. Now, it's a single thread (like keepalive) that periodically retrieves only contacts whose expiration time is < now and deletes them. A new contact_expiration_check_interval was added to global with a default of 30 seconds. Ross Beer reports that with this patch, his Asterisk startup time dropped from around an hour to under 30 seconds. There are still objects that can't be filtered at the database like identifies, transports, and registrations. These are not going to be anywhere near as numerous as endpoints, aors, auths, contacts however. Back to allow_unqualified_fetch. If this is set to yes and you have a very large number of objects in the database, the pjsip CLI commands will attempt to retrive ALL of them if not qualified with a LIKE. Worse, if you type "pjsip show endpoint <tab>" guess what's going to happen? :) Having a cache helps but all the objects will have to be retrieved at least once to fill the cache. Setting allow_unqualified_fetch=no prevents the mass retrieve and should be used on endpoints, auths, aors, and contacts. It should NOT be used for identifies, registrations and transports since these MUST be retrieved in bulk. Example sorcery.conf: [res_pjsip] endpoint=config,pjsip.conf,criteria=type=endpoint endpoint=realtime,ps_endpoints,allow_unqualified_fetch=error ASTERISK-25826 #close Reported-by: Ross Beer Tested-by: Ross Beer Change-Id: Id2691e447db90892890036e663aaf907b2dc1c67	9 years ago
Matt Jordan	0b6ff77afb	res/res_sorcery_astdb: Add a debugging message for when retrieval by ID fails ... Having a debug message tell us that we attempted to look up an item but failed is nice in circumstances when it isn't clear if the wizard was queried correctly or not. Change-Id: I2600c3bbea87f252196358f62e73f4c7da8632f7	10 years ago
Mark Michelson	dcf1ad14da	Add module support level to ast_module_info structure. Print it in CLI "module show" . ... ASTERISK-23919 #close Reported by Malcolm Davenport Review: https://reviewboard.asterisk.org/r/3802 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@419592 65c4cc65-6c06-0410-ace0-fbb531ad65f3	11 years ago
Richard Mudgett	d277f3ec3e	json: Fix off-nominal json ref counting issues. ... * Fixed off-nominal json ref counting issue with using the following API calls: ast_json_object_set() and ast_json_array_append(). * Fixed off-nominal error reporting in ast_ari_endpoints_list(). * Fixed some miscellaneous off-nominal json ref counting issues in report_receive_fax_status() and dial_to_json(). ........ Merged revisions 408713 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@408714 65c4cc65-6c06-0410-ace0-fbb531ad65f3	11 years ago
Richard Mudgett	61a4a98ed9	res_sorcery_astdb.c: Fix regex handling and keep simple prefix matching performance. ... The sorcery astDB wizzard does not handle regex correctly if the pattern begins with an anchor character. This patch attempts to convert the anchored regex pattern to a prefix pattern supported by astDB for performance reasons. If it is not able to convert the pattern it falls back to getting all astDB members of the family and doing a normal regex pattern matching on the retrieved records. Review: https://reviewboard.asterisk.org/r/3161/ ........ Merged revisions 408385 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@408386 65c4cc65-6c06-0410-ace0-fbb531ad65f3	11 years ago
Richard Mudgett	c91d4d12de	Reverting regex part of -r403545 at request of file. ... res_sorcery_astdb.c: Fix get multiple records by regex. * Fix sorcery_astdb_retrieve_regex() pattern matching. Let the regexec() function match the stored key values instead of having astdb prefilter them. Previoiusly you could only use a simple regex pattern when the pattern began with '^'. ........ Merged revisions 403559 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@403560 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
Richard Mudgett	31d51d373b	res_sorcery_astdb.c: Fix get multiple records by regex. ... * Fix sorcery_astdb_retrieve_regex() pattern matching. Let the regexec() function match the stored key values instead of having astdb prefilter them. Previoiusly you could only use a simple regex pattern when the pattern began with '^'. * Fix off nominal memory leak in sorcery_astdb_retrieve_regex(). ........ Merged revisions 403545 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@403546 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
David M. Lee	10c91bc96e	Address JSON thread safety issues. ... In tracking down some unit tests failures, I ended up reading the fine print[1] regarding Jansson's thread safety. In short: 1. Ref-counting is non-atomic. 2. json_dumps() and friends are not thread safe. This patch adds locking where necessary to our ast_json_* wrapper API, with documentation in json.h describing the thread safety limitations of the API. [1]: http://www.digip.org/jansson/doc/2.4/portability.html#thread-safety Review: https://reviewboard.asterisk.org/r/2716/ git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@396119 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
Joshua Colp	40074542bf	Add support for observers and JSON objectset creation to sorcery. ... This change adds the ability for modules to add themselves as observers to sorcery object types. Observers can be notified when objects are created, updated, or deleted as well as when the object type is loaded or reloaded. Observer notifications are done using a thread pool in a serialized fashion so the caller of the sorcery API calls is minimally impacted. This also adds the ability to create JSON changesets of a sorcery object. Tests are also present to confirm all of the above functionality. Review: https://reviewboard.asterisk.org/r/2477/ git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@387662 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
Joshua Colp	426095bc55	Add a res_sorcery_astdb module which uses the astdb to persist objects. ... Review: https://reviewboard.asterisk.org/r/2420/ git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@384857 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago