11 Commits (60cc53ecb3fd92efbc6a37f86caeb94d29e9eb38)

Author	SHA1	Message	Date
George Joseph	5aa5c49413	sorcery/res_pjsip: Refactor for realtime performance ... There were a number of places in the res_pjsip stack that were getting all endpoints or all aors, and then filtering them locally. A good example is pjsip_options which, on startup, retrieves all endpoints, then the aors for those endpoints, then tests the aors to see if the qualify_frequency is > 0. One issue was that it never did anything with the endpoints other than retrieve the aors so we probably could have skipped a step and just retrieved all aors. But nevermind. This worked reasonably well with local config files but with a realtime backend and thousands of objects, this was a nightmare. The issue really boiled down to the fact that while realtime supports predicates that are passed to the database engine, the non-realtime sorcery backends didn't. They do now. The realtime engines have a scheme for doing simple comparisons. They take in an ast_variable (or list) for matching, and the name of each variable can contain an operator. For instance, a name of "qualify_frequency >" and a value of "0" would create a SQL predicate that looks like "where qualify_frequency > '0'". If there's no operator after the name, the engines add an '=' so a simple name of "qualify_frequency" and a value of "10" would return exact matches. The non-realtime backends decide whether to include an object in a result set by calling ast_sorcery_changeset_create on every object in the internal container. However, ast_sorcery_changeset_create only does exact string matches though so a name of "qualify_frequency >" and a value of "0" returns nothing because the literal "qualify_frequency >" doesn't match any name in the objset set. So, the real task was to create a generic string matcher that can take a left value, operator and a right value and perform the match. To that end, strings.c has a new ast_strings_match(left, operator, right) function. Left and right are the strings to operate on and the operator can be a string containing any of the following: = (or NULL or ""), !=, >, >=, <, <=, like or regex. If the operator is like or regex, the right string should be a %-pattern or a regex expression. If both left and right can be converted to float, then a numeric comparison is performed, otherwise a string comparison is performed. To use this new function on ast_variables, 2 new functions were added to config.c. One that compares 2 ast_variables, and one that compares 2 ast_variable lists. The former is useful when you want to compare 2 ast_variables that happen to be in a list but don't want to traverse the list. The latter will traverse the right list and return true if all the variables in it match the left list. Now, the backends' fields_cmp functions call ast_variable_lists_match instead of ast_sorcery_changeset_create and they can now process the same syntax as the realtime engines. The realtime backend just passes the variable list unaltered to the engine. The only gotcha is that there's no common realtime engine support for regex so that's been noted in the api docs for ast_sorcery_retrieve_by_fields. Only one more change to sorcery was done... A new config flag "allow_unqualified_fetch" was added to reg_sorcery_realtime. "no": ignore fetches if no predicate fields were supplied. "error": same as no but emit an error. (good for testing) "yes": allow (the default); "warn": allow but emit a warning. (good for testing) Now on to res_pjsip... pjsip_options was modified to retrieve aors with qualify_frequency > 0 rather than all endpoints then all aors. Not only was this a big improvement in realtime retrieval but even for config files there's an improvement because we're not going through endpoints anymore. res_pjsip_mwi was modified to retieve only endpoints with something in the mailboxes field instead of all endpoints then testing mailboxes. res_pjsip_registrar_expire was completely refactored. It was retrieving all contacts then setting up scheduler entries to check for expiration. Now, it's a single thread (like keepalive) that periodically retrieves only contacts whose expiration time is < now and deletes them. A new contact_expiration_check_interval was added to global with a default of 30 seconds. Ross Beer reports that with this patch, his Asterisk startup time dropped from around an hour to under 30 seconds. There are still objects that can't be filtered at the database like identifies, transports, and registrations. These are not going to be anywhere near as numerous as endpoints, aors, auths, contacts however. Back to allow_unqualified_fetch. If this is set to yes and you have a very large number of objects in the database, the pjsip CLI commands will attempt to retrive ALL of them if not qualified with a LIKE. Worse, if you type "pjsip show endpoint <tab>" guess what's going to happen? :) Having a cache helps but all the objects will have to be retrieved at least once to fill the cache. Setting allow_unqualified_fetch=no prevents the mass retrieve and should be used on endpoints, auths, aors, and contacts. It should NOT be used for identifies, registrations and transports since these MUST be retrieved in bulk. Example sorcery.conf: [res_pjsip] endpoint=config,pjsip.conf,criteria=type=endpoint endpoint=realtime,ps_endpoints,allow_unqualified_fetch=error ASTERISK-25826 #close Reported-by: Ross Beer Tested-by: Ross Beer Change-Id: Id2691e447db90892890036e663aaf907b2dc1c67	10 years ago
Mark Michelson	32fc784284	res_sorcery_realtime: Fix regex regression. ... A regression was introduced where searching for realtime PJSIP objects by regex by starting the regex with a leading "^" would cause no items to be returned. This was due to a change which attempted to drop the requirement for a leading "^" to be present due to how some CLI commands formulate their regexes. However, the change, rather than simply eliminating the requirement, caused any regexes that did begin with "^" to end up not returning the expected results. This change fixes the problem by inspecting the regex and formulating the realtime query differently depending on if it begins with "^". ASTERISK-25702 #close Reported by Nic Colledge Patches: realtime_retrieve_regex.patch submitted by Alexei Gradinari License #5691 Change-Id: I055df608a6e6a10732044fa737a9fe8dca602693	10 years ago
Mark Michelson	4cd58c3b20	res_sorcery_realtime: Remove leading ^ requirement. ... res_sorcery_realtime's search-by-regex callback performed a check to ensure that the passed-in regex began with a caret (^). If it did not, then no results would be returned. This callback only started to become used when "like" support was added to PJSIP CLI commands. The CLI command for listing objects would pass an empty regex ("") to the sorcery backend if no "like" statement was present. For most sorcery backends, this resulted in returning all objects. However, for realtime, this resulted in returning no objects. This commit seeks to fix the regression by removing the requirement from res_sorcery_realtime for the passed-in-regex to begin with a caret. ASTERISK-25689 #close Reported by Marcelo Terres Change-Id: I22b4dc5d7f3f11bb29ac2e42ef94682e9bab3b20	10 years ago
Richard Mudgett	fc45f4040d	res_sorcery_realtime.c: Fix crash from NULL sorcery object type. ... If the sorcery object type is not found a NULL is returned. Unfortunately, sorcery_realtime_filter_objectset() will crash after complaining about not finding the object type and saying to expect errors. * Use ao2_cleanup() instead of ao2_ref() to prevent the crash. ASTERISK-25165 Reported by Corey Farrell Change-Id: Ic3b64453ea3058cb68d5c26d97d4fe7b8eea2e97	10 years ago
Mark Michelson	156395e743	res_sorcery_realtime: Fix leak of sorcery object type. ... This prevents a leak of a sorcery object type when realtime sorcery objects are retrieved by fields or when multiple objects are retrieved. The extent of this leak is that sorcery object types would be leaked. These are allocated whenever an object type is registered with sorcery, meaning that on module shutdown, these objects would be leaked. This could be problematic if many reloads were performed, but it is not as severe as if every sorcery object retrieved from realtime were being leaked. ASTERISK-25165 #close Reported by Corey Farrell Change-Id: I625c3b50eee4576670b7eeb013c81ad043b4b4f8	10 years ago
Richard Mudgett	dea885a607	A couple minor cleanup tweaks. ... * In res/res_sorcery_realtime.c: Broke long line. * In main/bucket.c: Eliminated unnecessary NULL check as ast_sorcery_unref() is NULL tolerant and set the global object to NULL after unref in the system shutdown bucket_cleanup(). git-svn-id: https://origsvn.digium.com/svn/asterisk/branches/13@433420 65c4cc65-6c06-0410-ace0-fbb531ad65f3	11 years ago
George Joseph	8bb06d592a	manager/config: Support templates and non-unique category names via AMI ... This patch provides the capability to manipulate templates and categories with non-unique names via AMI. Summary of changes: GetConfig and GetConfigJSON: Added "Filter" parameter: A comma separated list of name_regex=value_regex expressions which will cause only categories whose variables match all expressions to be considered. The special variable name TEMPLATES can be used to control whether templates are included. Passing 'include' as the value will include templates along with normal categories. Passing 'restrict' as the value will restrict the operation to ONLY templates. Not specifying a TEMPLATES expression results in the current default behavior which is to not include templates. UpdateConfig: NewCat now includes options for allowing duplicate category names, indicating if the category should be created as a template, and specifying templates the category should inherit from. The rest of the actions now accept a filter string as defined above. If there are non-unique category names, you can now update specific ones based on variable values. To facilitate the new capabilities in manager, corresponding changes had to be made to config, most notably the addition of filter criteria to many of the APIs. In some cases it was easy to change the references to use the new prototype but others would have required touching too many files for this patch so a wrapper with the original prototype was created. Macros couldn't be used in this case because it would break binary compatibility with modules such as res_digium_phone that are linked to real symbols. Tested-by: George Joseph Review: https://reviewboard.asterisk.org/r/4033/ ........ Merged revisions 425383 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/branches/13@425384 65c4cc65-6c06-0410-ace0-fbb531ad65f3	11 years ago
Mark Michelson	dcf1ad14da	Add module support level to ast_module_info structure. Print it in CLI "module show" . ... ASTERISK-23919 #close Reported by Malcolm Davenport Review: https://reviewboard.asterisk.org/r/3802 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@419592 65c4cc65-6c06-0410-ace0-fbb531ad65f3	11 years ago
Mark Michelson	9665c2d3eb	Handle the return values of realtime updates and stores more accurately. ... Realtime backends' update and store callbacks return the number of rows affected, or -1 if there was a failure. There were a couple of issues: * The config API was treating 0 as a successful return, and positive values as a failure. Now the config API treats anything >= 0 as a success. * res_sorcery_realtime was treating 0 as a successful return from the store procedure, and any positive values as a failure. Now sorcery treats anything > 0 as a success. It still considers 0 a "failure" since there is no change to report to observers. Review: https://reviewboard.asterisk.org/r/3341 ........ Merged revisions 410592 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@410593 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
Mark Michelson	c162101d69	Make res_sorcery_realtime filter unknown retrieved results. ... When retrieving data from a database or other realtime backend, it's quite possible to retrieve variables that Asterisk does not care about but that are legitimate to exist. Asterisk does not need to throw a hissy fit when these variables are encountered but rather just filter them out. Review: https://reviewboard.asterisk.org/r/3305 ........ Merged revisions 410187 from http://svn.asterisk.org/svn/asterisk/branches/12 git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@410207 65c4cc65-6c06-0410-ace0-fbb531ad65f3	12 years ago
Joshua Colp	02be50b1ac	Add support for a realtime sorcery module. ... This change does the following: 1. Adds the sorcery realtime module 2. Adds unit tests for the sorcery realtime module 3. Changes the realtime core to use an ast_variable list instead of variadic arguments 4. Changes all realtime drivers to accept an ast_variable list Review: https://reviewboard.asterisk.org/r/2424/ git-svn-id: https://origsvn.digium.com/svn/asterisk/trunk@386731 65c4cc65-6c06-0410-ace0-fbb531ad65f3	13 years ago